Linux – Performance Issues Troubleshooting

Key System resources in Redhat Linux:
1. CPU 
2. Memory
3. Swap
4. Filesystem (Disk or LUN)
5. Network

1.CPU:

CPU utilization can be monitor using various in build Linux tools. top,vmstat  and “sar -u” are few of them.You can also check out here about CPU information. 

To get the current CPU utilization details,

VMSTAT

[root@Global-RH ~]# vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 402432  45308 449980    0    0    92    54   74  112  1  2 94  3  0
 0  0      0 402424  45308 450008    0    0     0     0   54   82  1  0 99  0  0
 0  0      0 402424  45308 450008    0    0     0     0   33   52  0  0 100  0  0
 0  0      0 402432  45316 450004    0    0     0    72   59  109  0  1 99  0  0
 0  0      0 402432  45316 450008    0    0     0     0   37   59  0  0 100  0  0

SAR

[root@Global-RH ~]# sar -u 5 5
Linux 2.6.32-279.el6.x86_64 (Global-RH)         11/25/2013      _x86_64_        (1 CPU)

10:34:11 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:34:16 AM     all      0.00      0.00      0.80      0.00      0.00     99.20
10:34:21 AM     all      0.40      0.00      2.00      0.00      0.00     97.60
10:34:26 AM     all      0.00      0.00      1.41      0.00      0.00     98.59
10:34:31 AM     all      0.00      0.00      0.40      0.00      0.00     99.60
10:34:36 AM     all      0.00      0.00      0.80      1.00      0.00     98.20
Average:        all      0.08      0.00      1.08      0.20      0.00     98.63
[root@Global-RH ~]#

TOP

top - 10:36:09 up  1:16,  4 users,  load average: 0.00, 0.01, 0.04
Tasks: 156 total,   1 running, 155 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.3%us,  1.5%sy,  0.0%ni, 93.9%id,  3.3%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   1250268k total,   841816k used,   408452k free,    45384k buffers
Swap:  2523128k total,        0k used,  2523128k free,   450152k cached

High CPU consuming process:

[root@Global-RH ~]# top
top - 10:42:11 up  1:22,  4 users,  load average: 0.76, 0.22, 0.08
Tasks: 156 total,   2 running, 154 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1250268k total,   889884k used,   360384k free,    80212k buffers
Swap:  2523128k total,        0k used,  2523128k free,   450164k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6010 root      20   0  109m 1244  888 R 98.1  0.1   0:32.21 find
 1094 root      20   0     0    0    0 S  1.3  0.0   0:02.40 flush-253:0
 6031 root      20   0 15028 1296  964 R  0.7  0.1   0:00.03 top
 2680 root      20   0 40336  616  364 S  0.3  0.0   0:01.20 udisks-daemon
    1 root      20   0 19348 1564 1252 S  0.0  0.1   0:02.07 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.01 kthreadd

Using PS command,

[root@Global-RH ~]# ps -eo pcpu,args | sort -k 1 -r | head -8
%CPU COMMAND
90.5 find / -name temp_myname*
 0.2 /usr/sbin/vmtoolsd
 0.2 /usr/lib/vmware-tools/sbin64/vmtoolsd -n vmusr --blockFd 3
 0.1 /usr/sbin/restorecond -u
 0.0 [watchdog/0]
 0.0 [vmmemctl]
 0.0 /usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -B -u -f /var/log/wpa_supplicant.log -P /var/run/wpa_supplicant.pid
[root@Global-RH ~]#

2.Memory:

Memory bottle can be easily identified using vmstat and sar command. You need to be careful to determine the free memory.Because redhat Linux will use free physical memory as cache. The cache memory will be released when its required for applications.

To get the memory information,

Using Meminfo,

[root@Global-RH ~]# cat /proc/meminfo
MemTotal:        1250268 kB
MemFree:          408724 kB
Buffers:           45416 kB
Cached:           450156 kB
SwapCached:            0 kB
Active:           319124 kB
Inactive:         333028 kB
Active(anon):     156764 kB
Inactive(anon):     3332 kB
Active(file):     162360 kB
Inactive(file):   329696 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2523128 kB
SwapFree:        2523128 kB
Dirty:                28 kB
Writeback:             0 kB
AnonPages:        156596 kB
Mapped:            71844 kB
Shmem:              3520 kB
Slab:             124828 kB
SReclaimable:      63844 kB
SUnreclaim:        60984 kB
KernelStack:        2040 kB
PageTables:        27704 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3148260 kB
Committed_AS:     712176 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      280420 kB
VmallocChunk:   34359440948 kB
HardwareCorrupted:     0 kB
AnonHugePages:     28672 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8192 kB
DirectMap2M:     1282048 kB
[root@Global-RH ~]#

Using stat and sar commands,

[root@Global-RH ~]# vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 4  0      0 326304  80864 450164    0    0    85    52  119  126  1  6 89  3  0
 5  0      0 326272  80864 450164    0    0     0     0 1021  587  0 100  0  0  0
 5  0      0 326140  80864 450164    0    0     0     0 1017  700  0 100  0  0  0
 6  0      0 326272  80864 450164    0    0     0     0 1018  671  0 100  0  0  0
 5  0      0 326272  80864 450164    0    0     0     0 1019  658  0 100  0  0  0

Using SAR command,

[root@Global-RH ~]# sar -r 5 5
Linux 2.6.32-279.el6.x86_64 (Global-RH)         11/25/2013      _x86_64_        (1 CPU)

10:48:22 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
10:48:27 AM    326404    923864     73.89     80872    450164    728784     19.31
10:48:32 AM    326388    923880     73.89     80872    450164    728784     19.31
10:48:37 AM    326404    923864     73.89     80872    450164    728784     19.31
10:48:42 AM    327140    923128     73.83     80872    450164    727352     19.28
10:48:47 AM    327388    922880     73.81     80872    450164    727352     19.28
Average:       326745    923523     73.87     80872    450164    728211     19.30

Using top command,

[root@Global-RH ~]# top
top - 10:49:04 up  1:29,  4 users,  load average: 5.69, 2.87, 1.18
Tasks: 162 total,   7 running, 155 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1250268k total,   922616k used,   327652k free,    80880k buffers
Swap:  2523128k total,        0k used,  2523128k free,   450164k cached

Using Free command,

[root@Global-RH ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          1220        897        323          0         79        439
-/+ buffers/cache:        379        841
Swap:         2463          0       2463
[root@Global-RH ~]#

As per the above commands outputs , system has 1220MB physical memory and used memory is 897MB. Free memory is 323MB. In that used memory (897MB), 439MB is using as cache by the system.This memory will be released upon the application demands. 

If you are running out of free memory and cached memory is very less ,then system is in real memory bottleneck.

High memory consuming process,
Using top,

[root@Global-RH ~]# top -M
top - 10:56:52 up  1:37,  4 users,  load average: 0.01, 0.97, 1.01
Tasks: 155 total,   1 running, 154 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  1220.965M total,  897.832M used,  323.133M free,   79.102M buffers
Swap: 2463.992M total,    0.000k used, 2463.992M free,  439.621M cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    7 root      20   0     0    0    0 S  0.3  0.0   0:04.16 events/0
 2706 root      20   0  426m  26m  18m S  0.3  2.1   0:15.30 vmtoolsd
 6426 root      20   0 15028 1284  964 R  0.3  0.1   0:00.08 top
    1 root      20   0 19348 1564 1252 S  0.0  0.1   0:02.08 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.02 kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0

using ps command,

[root@Global-RH ~]# ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS |tail -10
13088 gnome-volume-control-applet
13268 /usr/bin/gnome-terminal -x /bin/sh -c cd '/root/Desktop' && exec $SHELL
13656 /usr/libexec/clock-applet --oaf-activate-iid=OAFIID:GNOME_ClockApplet_Factory --oaf-ior-fd=28
14444 nm-applet --sm-disable
17796 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=22
19260 python /usr/share/system-config-printer/applet.py
20016 /usr/sbin/restorecond -u
20628 nautilus
25744 /usr/bin/Xorg :0 -nr -verbose -audit 4 -auth /var/run/gdm/auth-for-gdm-0wmzZs/database -nolisten tcp vt1
26824 /usr/lib/vmware-tools/sbin64/vmtoolsd -n vmusr --blockFd 3
[root@Global-RH ~]#

Using ps command with command details,

[root@Global-RH ~]# /bin/ps ax -orss,%mem,cmd --sort=rss|tac|head -10
26824  2.1 /usr/lib/vmware-tools/sbin64/vmtoolsd -n vmusr --blockFd 3
25744  2.0 /usr/bin/Xorg :0 -nr -verbose -audit 4 -auth /var/run/gdm/auth-for-gdm-0wmzZs/database -nolisten tcp vt1
20628  1.6 nautilus
20016  1.6 /usr/sbin/restorecond -u
19260  1.5 python /usr/share/system-config-printer/applet.py
17796  1.4 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=22
14444  1.1 nm-applet --sm-disable
13656  1.0 /usr/libexec/clock-applet --oaf-activate-iid=OAFIID:GNOME_ClockApplet_Factory --oaf-ior-fd=28
13268  1.0 /usr/bin/gnome-terminal -x /bin/sh -c cd '/root/Desktop' && exec $SHELL
13088  1.0 gnome-volume-control-applet
[root@Global-RH ~]#

3.Swap:

When the physical memory is completely used,then system will be start using the swap space.If the system is running out of swap space, you can see fork errors in the /var/log/messages file. If the system is having 2GB physical memory and configuring with 8GB swap is completely  waste. If system start swapping more process to the disk, system performance will be degrade.

I can find very few commands to list the swap information.
1.swapfs

[root@Global-RH ~]# cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       2523128 0       -1
[root@Global-RH ~]#

2.Free command.

[root@Global-RH ~]# free -t
             total       used       free     shared    buffers     cached
Mem:       1250268     920380     329888          0      81248     450584
-/+ buffers/cache:     388548     861720
Swap:      2523128          0    2523128
Total:     3773396     920380    2853016
[root@Global-RH ~]#

3.Top

[root@Global-RH ~]# top
top - 11:12:37 up  1:53,  4 users,  load average: 0.00, 0.03, 0.34
Tasks: 156 total,   1 running, 155 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1250268k total,   920388k used,   329880k free,    81272k buffers
Swap:  2523128k total,        0k used,  2523128k free,   450584k cached

How to identify the memory/swap bottle neck ? 
unfortunately Linux doesn’t offer to get the scan rate(sr) to identify the high swap rate like Unix.You need to look at swap in (si) and swap out (so) rate in vmstat command output to determine that. 

4. Filesystem (Disk/LUN I/O Bottle Neck)

Sometimes you can see that system may have enough free CPU and Memory resources but still see some performance issues.In these cases ,you need to look at the “iowait”  field in “iostat -x” . “mpstat” also you can see the iowait. In Top command, you need to look at “wa%” field. If anything more than 10 ,then CPU’s are waiting for disk to complete the write. Most of the time SAN’s poor performance will make the iowait value to higher. 

Using Mpstat,

[root@Global-RH ~]# mpstat 1 5
Linux 2.6.32-279.el6.x86_64 (Global-RH)         11/25/2013      _x86_64_        (1 CPU)

11:38:50 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
11:38:51 AM  all    0.00    0.00  100.00   46.67    0.00    0.00    0.00    0.00    0.00
11:38:52 AM  all    0.00    0.00  100.00   89.67    0.00    0.00    0.00    0.00    0.00
11:38:53 AM  all    0.00    0.00  100.00   94.67    0.00    0.00    0.00    0.00    0.00
11:38:54 AM  all    0.00    0.00   83.33   46.67    0.00    0.00    0.00    0.00    0.00
11:38:55 AM  all    4.35    0.00   95.65   63.67    0.00    0.00    0.00    0.00    0.00
Average:     all    1.72    0.00   94.83   99.67    0.00    0.00    0.00    0.00    0.00
[root@Global-RH ~]#

Using iostat,

[root@Global-RH ~]# iostat -x
Linux 2.6.32-279.el6.x86_64 (Global-RH)         11/25/2013      _x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.99    0.01    6.29    2.26    0.00   90.45

Device: rrqm/s   wrqm/s   r/s     w/s     rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
scd0     0.23     0.00    0.06    0.00     1.14     0.00    19.95     0.00    4.85   3.64   0.02
sda      0.78     8.01    3.09    2.65   116.94    85.50    35.33     0.52   90.15   5.61   93.21
dm-0     0.00     0.00    3.70   10.69   115.76    85.50    13.99     2.29  159.42   2.23   97.21
dm-1     0.00     0.00    0.04    0.00     0.32     0.00     8.00     0.00    4.09   2.38   0.01
[root@Global-RH ~]#

If the svctm is less than 10 is acceptable value for SAN environment. If the svctm value is <10 and still the utilization is more than 60% ,then you need to tune the application or need to spread the database to multiple filesystem to increase the write rate.

How to identify which process is making high i/o wait on the system ?
Use the below syntax to find the process which is making high i/o wait to the system

[root@Global-RH ~]#  for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done
D  7161 cp -i -r /usr /usr_old
----
D   391 [jbd2/dm-0-8]
----
^C
---
[root@Global-RH ~]#

Now we need to get the details of the pid 7161 and 391.

[root@Global-RH ~]# cat /proc/7161/io
rchar: 145039848
wchar: 145032790
syscr: 14556
syscw: 8777
read_bytes: 87707648
write_bytes: 158609408
cancelled_write_bytes: 0
[root@Global-RH ~]# lsof -p 7161
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
cp      7161 root  cwd    DIR  253,0     4096 130821 /root
cp      7161 root  rtd    DIR  253,0     4096      2 /
cp      7161 root  txt    REG  253,0   122736    900 /bin/cp
cp      7161 root  mem    REG  253,0   156872 178706 /lib64/ld-2.12.so
cp      7161 root  mem    REG  253,0    22536 178723 /lib64/libdl-2.12.so
cp      7161 root  mem    REG  253,0  1918016 178707 /lib64/libc-2.12.so
cp      7161 root  mem    REG  253,0   145720 178710 /lib64/libpthread-2.12.so
cp      7161 root  mem    REG  253,0    47064 178711 /lib64/librt-2.12.so
cp      7161 root  mem    REG  253,0   124624 178741 /lib64/libselinux.so.1
cp      7161 root  mem    REG  253,0    33816 178838 /lib64/libacl.so.1.1.0
cp      7161 root  mem    REG  253,0    21152 150217 /lib64/libattr.so.1.1.0
cp      7161 root  mem    REG  253,0 99158576 134657 /usr/lib/locale/locale-archive
cp      7161 root    0u   CHR  136,2      0t0      5 /dev/pts/2
cp      7161 root    1u   CHR  136,2      0t0      5 /dev/pts/2
cp      7161 root    2u   CHR  136,2      0t0      5 /dev/pts/2
cp      7161 root    3r   REG  253,0 60073002 169255 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar
cp      7161 root    4w   REG  253,0 29884416 407742 /usr_old/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar
[root@Global-RH ~]#

In this way you can pinpoint the iowait. 

5.Network Bottle Neck.

Network overloading very rare case to impact the system performance.

Look at the interface for any errors(RX-ERR,TX-ERR) using netstat command.

[root@Global-RH ~]# netstat -i
Kernel Interface table
Iface   MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0    1500   0    10697      0      0      0     6930      0      0      0 BMRU
lo     16436   0       20      0      0      0       20      0      0      0 LRU
virbr0  1500   0        0      0      0      0       26      0      0      0 BMRU
[root@Global-RH ~]#

dstat is very important tool to monitor the all the system resources.If its not already installed,install it using “yum install dtstat”

[root@Global-RH ~]# dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  1   6  89   4   0   0| 126k  107k|   0     0 |   0     0 | 114   123
  0   0 100   0   0   0|   0     0 |  60B  826B|   0     0 |  62   113
  0   0 100   0   0   0|   0     0 | 166B  506B|   0     0 |  60   101
  0   1  99   0   0   0|   0     0 |  60B  346B|   0     0 |  57    97
  0   0 100   0   0   0|   0    12k|  60B  346B|   0     0 |  53    98
  0   0 100   0   0   0|   0     0 |  60B  346B|   0     0 |  51    95
  0   0 100   0   0   0|   0     0 | 152B  346B|   0     0 |  50    99
  0   1  99   0   0   0|   0     0 | 152B  346B|   0     0 |  61   100 ^C
[root@Global-RH ~]#

You can see the nework traffic details in “net/total” fields.

Here is the complete list of option for dstat tool.

[root@Global-RH ~]# dstat -h
Usage: dstat [-afv] [options..] [delay [count]]
Versatile tool for generating system resource statistics

Dstat options:
  -c, --cpu              enable cpu stats
     -C 0,3,total           include cpu0, cpu3 and total
  -d, --disk             enable disk stats
     -D total,hda           include hda and total
  -g, --page             enable page stats
  -i, --int              enable interrupt stats
     -I 5,eth2              include int5 and interrupt used by eth2
  -l, --load             enable load stats
  -m, --mem              enable memory stats
  -n, --net              enable network stats
     -N eth1,total          include eth1 and total
  -p, --proc             enable process stats
  -r, --io               enable io stats (I/O requests completed)
  -s, --swap             enable swap stats
     -S swap1,total         include swap1 and total
  -t, --time             enable time/date output
  -T, --epoch            enable time counter (seconds since epoch)
  -y, --sys              enable system stats

  --aio                  enable aio stats
  --fs, --filesystem     enable fs stats
  --ipc                  enable ipc stats
  --lock                 enable lock stats
  --raw                  enable raw stats
  --socket               enable socket stats
  --tcp                  enable tcp stats
  --udp                  enable udp stats
  --unix                 enable unix stats
  --vm                   enable vm stats

  --plugin-name          enable plugins by plugin name (see manual)
  --list                 list all available plugins

  -a, --all              equals -cdngy (default)
  -f, --full             automatically expand -C, -D, -I, -N and -S lists
  -v, --vmstat           equals -pmgdsc -D total

  --bw, --blackonwhite   change colors for white background terminal
  --float                force float values on screen
  --integer              force integer values on screen
  --nocolor              disable colors (implies --noupdate)
  --noheaders            disable repetitive headers
  --noupdate             disable intermediate updates
  --output file          write CSV output to file

delay is the delay in seconds between each update (default: 1)
count is the number of updates to display before exiting (default: unlimited)

[root@Global-RH ~]#

Hope you got some visibility to troubleshooting the performance issues on Redhat Linux .

Thank you for visiting UnixArena.