If its comes to performance issue in Unix operating systems, everybody will be behind on CPU.But not all the time CPU will cause for the performance problem. Because sometimes CPU may waiting for resources like Disk I/O, Swapping ,Network.As sysadmin you should able to pin point where the problem.vmstat is very useful command to determined where it the problem.
After introduction Solaris zones, sysadmin job is become bit difficult to isolate the problems from other local zones and global zone. prstat is very useful command to find local zones CPU usage from global zone.
After introduction Solaris zones, sysadmin job is become bit difficult to isolate the problems from other local zones and global zone. prstat is very useful command to find local zones CPU usage from global zone.
VMSTAT:
vmstat is very useful command to find the root cause for probing issues.last columns shows the CPU ideal status.The field which you need to look at in the vmstat are
r , b ,w,sr,cpu id
r , b ,w,sr,cpu id
bash-3.00# vmstat 1 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr f0 s0 s1 s2 in sy cs us sy id 0 0 0 1061088 86408 36 99 0 0 1 0 6 -0 2 -0 0 328 458 186 0 1 98 1 0 0 954432 93008 15 49 0 0 0 0 0 0 0 0 0 546 583 307 0 2 97 0 0 0 954352 92956 8 8 0 0 0 0 0 0 0 0 0 470 545 297 0 1 99 0 0 0 954352 92956 8 8 0 0 0 0 0 0 0 0 0 566 563 328 0 2 98 0 0 0 954352 92956 8 8 0 0 0 0 0 0 0 0 0 581 605 315 0 1 99 0 0 0 954352 92956 8 8 0 0 0 0 0 0 0 0 0 553 564 301 0 1 99 2 0 0 954352 92956 8 8 0 0 0 0 0 0 0 0 0 550 574 315 0 4 95 1 0 0 954352 92956 8 8 0 0 0 0 0 0 0 0 0 604 682 340 0 1 98 0 0 0 954352 92956 8 8 0 0 0 0 0 0 0 0 0 508 721 356 0 4 96 1 0 0 954352 92956 8 8 0 0 0 0 0 0 0 0 0 519 575 321 0 2 97
The fields of vmstat’s display are
procs
r in run queue
b blocked for resources I/O, paging etc.
w swapped
memory (in Kbytes)
swap – amount of swap space currently available
free – size of the free list page ( in units per second).
re page reclaims – see -S option for how this field is modified.
mf minor faults – see -S option for how this field is modified.
pi kilobytes paged in
po kilobytes paged out
fr kilobytes freed
de anticipated short-term memory shortfall (Kbytes)
sr pages scanned by clock algorithmdisk ( operations per second )
procs
r in run queue
b blocked for resources I/O, paging etc.
w swapped
memory (in Kbytes)
swap – amount of swap space currently available
free – size of the free list page ( in units per second).
re page reclaims – see -S option for how this field is modified.
mf minor faults – see -S option for how this field is modified.
pi kilobytes paged in
po kilobytes paged out
fr kilobytes freed
de anticipated short-term memory shortfall (Kbytes)
sr pages scanned by clock algorithmdisk ( operations per second )
There are slots for up to four disks, labeled with a single letter and number.
The letter indicates the type of disk (s = SCSI, i = IPI, etc) . The number is
the logical unit number. faults
in (non clock) device interrupts
sy system calls
cs CPU context switches cpu – breakdown of percentage usage of CPU time. On multiprocessors this is an a
in (non clock) device interrupts
sy system calls
cs CPU context switches cpu – breakdown of percentage usage of CPU time. On multiprocessors this is an a
verage across all processors.
us user time
sy system time
id idle time
Problem symptoms:
1.) If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs .
2.) if this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.
3.) If the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us) system is facing shortage of CPU resources.
Resolution :
Resolution to these kind of issues involves tuning of application procedures to make efficient use of cpu and as a last resort increasing the cpu power or adding more cpu to the system
Top:
Top is tool which is commonly available all the Unix/Linux.IN Solaris we need to install .To see top 10 process which are utilizing high CPU time,use below command.
bash-3.00# /usr/local/bin/top 10 last pid:5307;load avg:0.16, 1.25, 0.95; up 2+12:33:59 14:17:37 122 processes: 121 sleeping, 1 on cpu CPU states: 99.0% idle, 0.5% user, 0.5% kernel, 0.0% iowait, 0.0% swap Memory: 1024M phys mem, 91M free mem, 1124M total swap, 1054M free swap PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND 12321 root 1 59 0 67M 9688K sleep 3:07 0.17% mixer_applet2 5306 root 1 52 0 3104K 1696K cpu/1 0:00 0.12% top 12319 root 1 59 0 69M 11M sleep 1:19 0.07% gnome-netstatus 12251 root 1 59 0 10M 6100K sleep 1:16 0.06% gconfd-2 12226 root 1 59 0 25M 3596K sleep 1:15 0.06% vmtoolsd 19353 noaccess 18 59 0 122M 97M sleep 2:16 0.04% java 3961 noaccess 18 59 0 140M 87M sleep 6:17 0.03% java 3869 root 1 59 0 73M 20M sleep 1:33 0.03% Xorg 2015 root 1 59 0 7924K 3300K sleep 7:51 0.03% vmtoolsd 12300 root 1 59 0 72M 12M sleep 0:39 0.03% gnome-panel
Prstat:
Prstat is in-built Solaris performance monitoring tool.Its very useful to find zone wise CPU utilization and also we can monitor amount of time threads wait for a processor using prstat -mL command in the LAT coloum.
bash-3.00# prstat -a
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
18759 root 3856K 3144K cpu0 59 0 0:00:01 1.5% prstat/1
2015 root 7572K 3744K sleep 59 0 0:05:25 0.2% vmtoolsd/1
12226 root 25M 6696K sleep 59 0 0:00:28 0.2% vmtoolsd/1
18497 root 6448K 3644K sleep 59 0 0:00:02 0.2% sshd/1
12321 root 67M 10M sleep 59 0 0:00:52 0.2% mixer_applet2/1
5 root 0K 0K sleep 99 -20 0:03:47 0.2% zpool-rpool/39
12319 root 69M 13M sleep 59 0 0:00:23 0.1% gnome-netstatus/1
3961 noaccess 140M 95M sleep 59 0 0:04:10 0.1% java/18
12251 root 10M 7452K sleep 59 0 0:00:25 0.1% gconfd-2/1
3869 root 73M 24M sleep 59 0 0:01:00 0.0% Xorg/1
12300 root 72M 14M sleep 59 0 0:00:13 0.0% gnome-panel/1
18508 root 2984K 1716K sleep 59 0 0:00:00 0.0% bash/1
392 root 6156K 2776K sleep 59 0 0:00:07 0.0% nscd/33
NPROC USERNAME SWAP RSS MEMORY TIME CPU
84 root 172M 207M 20% 0:14:20 2.8%
1 noaccess 134M 101M 9.9% 0:04:10 0.1%
7 daemon 7528K 6816K 0.7% 0:00:12 0.0%
Total: 92 processes, 356 lwps, load averages: 0.08, 0.05, 0.04
bash-3.00# prstat -Z
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
4901 root 3808K 3052K cpu1 0 0 0:00:00 1.3% prstat/1
4879 root 6448K 3876K sleep 59 0 0:00:02 0.4% sshd/1
2015 root 7908K 3796K sleep 59 0 0:07:45 0.2% vmtoolsd/1
5 root 0K 0K sleep 99 -20 0:05:30 0.2% zpool-rpool/39
12321 root 67M 9892K sleep 59 0 0:03:04 0.2% mixer_applet2/1
19353 noaccess 122M 105M sleep 59 0 0:02:13 0.1% java/18
3961 noaccess 140M 94M sleep 59 0 0:06:14 0.1% java/18
12319 root 69M 12M sleep 59 0 0:01:15 0.1% gnome-netstatus/1
4890 root 3000K 1852K sleep 54 0 0:00:00 0.1% bash/1
12226 root 25M 4480K sleep 59 0 0:01:11 0.1% vmtoolsd/1
12251 root 10M 6684K sleep 59 0 0:01:15 0.1% gconfd-2/1
3869 root 73M 21M sleep 59 0 0:01:26 0.0% Xorg/1
12300 root 72M 13M sleep 59 0 0:00:38 0.0% gnome-panel/1
398 daemon 4260K 2212K sleep 59 0 0:00:02 0.0% kcfd/3
1889 root 3888K 1172K sleep 59 0 0:00:01 0.0% sshd/1
4878 root 4800K 2640K sleep 59 0 0:00:00 0.0% sshd/1
18497 root 6448K 3140K sleep 59 0 0:00:12 0.0% sshd/1
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
0 95 318M 242M 24% 0:32:03 2.7% global
10 30 157M 141M 14% 0:03:04 0.1% sol1
bash-3.00# prstat -mL
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
27258 root 4.2 17 0.0 0.0 0.0 0.0 78 0.2 76 0 48K 0 prstat/1
1907 root 0.4 0.6 0.0 0.1 0.0 0.0 99 0.3 51 2 2K 0 vmtoolsd/1
21975 root 0.2 0.1 0.0 0.0 0.0 0.0 100 0.1 30 1 193 0 mixer_applet/1
21291 root 0.2 0.0 0.0 0.1 0.1 0.0 100 0.2 51 0 212 0 vmtoolsd/1
21971 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.1 10 0 61 0 gnome-netsta/1
22001 root 0.1 0.1 0.0 0.0 0.0 98 2.2 0.1 6 0 94 0 firefox-bin/18
21289 root 0.1 0.1 0.0 0.0 0.0 0.0 100 0.1 20 1 161 0 gconfd-2/1
3803 noaccess 0.1 0.1 0.0 0.0 0.0 0.0 99 0.4 97 0 96 0 java/14
24052 root 0.1 0.1 0.0 0.0 0.1 0.0 99 0.8 11 0 42 0 gnome-termin/1
3803 noaccess 0.1 0.0 0.0 0.1 0.0 0.1 100 0.1 0 0 33 0 java/35
13099 root 0.1 0.0 0.0 0.0 0.1 0.0 100 0.1 25 0 100 11 Xorg/1
21583 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.1 20 0 124 0 gnome-panel/1
5 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 1 0 0 0 zpool-rpool/13
5 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 1 0 0 0 zpool-rpool/39
22001 root 0.0 0.0 0.0 0.3 0.3 0.0 99 0.0 3 0 14 0 firefox-bin/1
3803 noaccess 0.0 0.0 0.0 0.0 0.0 100 0.0 0.1 5 0 5 0 java/7
1597 daemon 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 4 0 nfsmapid/4
5 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 2 0 0 0 zpool-rpool/12
67 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 7 0 vxconfigd/1
5 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 2 0 0 0 zpool-rpool/38
5 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 1 0 0 0 zpool-rpool/28
5 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 1 0 0 0 zpool-rpool/19
5 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 1 0 0 0 zpool-rpool/31
5 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.1 1 0 0 0 zpool-rpool/30
5 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.1 1 0 0 0 zpool-rpool/27
5 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.1 1 0 0 0 zpool-rpool/21
24884 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 2 0 sshd/1
SAR:
The below command will CPU statistics for every 5 seconds for five times.We can add SAR in crontab to monitor CPU usage all the time.The load averages can be monitored extended time periods by looking at run queue lengths and the amount of time that the run queue is occupied using sar -q. High number of system calls per second per processor can be monitored using sar -c.
bash-3.00# sar -u 5 5 SunOS sfos 5.10 Generic_142910-17 i86pc 07/18/2012 14:05:40 usr %sys %wio %idle 14:05:45 0 3 0 97 14:05:50 1 2 0 98 14:05:55 0 2 0 97 14:06:00 1 3 0 96 14:06:05 0 2 0 98 Average 0 2 0 97 bash-3.00# sar -q SunOS sfos 5.10 Generic_142910-17 i86pc 04/29/2013 10:37:41 runq-sz %runocc swpq-sz %swpocc 10:40:01 1.3 24 39.0 100 10:45:00 1.3 19 39.0 100 10:50:01 1.0 18 39.0 100 Average 1.2 20 39.0 100 bash-3.00# sar -c SunOS sfos 5.10 Generic_142910-17 i86pc 04/29/2013 10:37:41 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s 10:40:01 933 115 60 0.16 0.15 11689 8178 10:45:00 886 68 32 0.78 0.77 113195 8021 10:50:01 744 45 21 0.40 0.40 58097 6056 Average 837 68 33 0.51 0.50 71549 7254
MPSTAT:
bash-3.00# mpstat CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 78 0 1 352 117 168 1 6 2 0 311 1 3 0 97 1 71 0 1 127 29 158 2 6 3 0 328 1 2 0 97
Thank you for reading this article.Please leave a comment if you have any doubt ,i will get back to you as soon as possible.
panini says
Great Article… Very useful… It would be great if you can provide explanation of all the possible terms to look at and what they mean.
Unix SysAdmin says
excellent thanks