Solaris has number of tools to examine process details and those just reading the /proc directory. These tools are very useful in application troubleshooting and analyzing the system performance. This article is based on “Solaris™ Internals: Solaris 10 and OpenSolaris Kernel Architecture” book. I would strongly recommend this for all the Solaris administrators.
Here we have differentiated the process stuffs in five Categories.
1.Process status tools (Ex:prstat,ps)
2.Process controlling tools (Ex:pkill,prun)
3.Process inspection tools (Ex:pstack,pfiles)
4.Process Lock Activity examination tools (Ex:plockstat)
5.Process Tracing tools (Ex:Dtrace,apptrace,truss)
The prstat statistics utility shows a top-level summary of the processes that are using system resources currently.The prstat utility summarizes this information for an every 5 seconds by default and reports the statistics for that period.
# prstat
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
13150 root 6448K 3744K sleep 39 0 0:00:01 0.3% sshd/1
5 root 0K 0K sleep 99 -20 0:38:00 0.3% zpool-rpool/39
1921 root 6708K 4064K sleep 59 0 0:02:19 0.1% vmtoolsd/1
413 daemon 4260K 2384K sleep 59 0 0:00:02 0.1% kcfd/3
13170 root 3744K 2920K cpu0 19 0 0:00:00 0.1% prstat/1
3436 root 11M 7532K sleep 59 0 0:00:14 0.1% dtgreet/1
13161 root 3008K 1772K sleep 49 0 0:00:00 0.1% bash/1
3374 root 61M 15M sleep 59 0 0:00:18 0.1% Xorg/1
9 root 12M 10M sleep 59 0 0:00:02 0.1% svc.startd/13
1814 root 3888K 1336K sleep 59 0 0:00:00 0.0% sshd/1
1611 daemon 4728K 1364K sleep 59 0 0:00:18 0.0% nfsmapid/4
The below table talks about default fields in prstat command output and its description.
| PRSTAT FIELDS | |
|---|---|
| Field | Description |
| PID | The unique process ID of the process |
| USERNAME | Owner (user or uid) of the process |
| SIZE | The total virtual memory size of mappings within the process, including all mapped files and devices. (Physical + Swap ) |
| RSS | Resident set size. The amount of physical memory mapped into the process,including that shared with other processes. |
| STATE | The state of the process |
| PRI | The priority of the process. Larger numbers mean higher priority. |
| NICE | Nice value used in priority computation. |
The below table explains about various prstat options.
| PRSTAT OPTIONS | |
|---|---|
| Options | Description |
| prstat | This shows a top-level summary of the processes that are using system resources currently for every five mins |
| prstat -L | This shows thread per line instead of one process per line.In last column you can see name of the process (executed file) |
| prstat -mL | It shows microstates of process & sow per-thread ( -L) |
| prstat -s | prstat ouput can be sorted using set of sub-options .sub options are cpu,pri,rss,size,time (Ex:prstat -s cpu) |
| cpu – sort by process cpu usage on the system | |
| pri – sort the process by priority | |
| rss – sort by resident set size (i.e physical memory) | |
| size – sort by size of the process (i.e virtual memory) | |
| time – sort by process execution time | |
| prstat -t | It provides complete users resource utlization |
| prstat -J | It gives project wise resource utilization |
| prstat -Z | It provides summary per local zone. |
Here we will see one by one prstat output
prstat -L
# prstat -L
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
13150 root 6448K 3744K sleep 59 0 0:00:01 0.2% sshd/1
1611 daemon 4728K 1364K sleep 59 0 0:00:17 0.1% nfsmapid/3
1921 root 6708K 4064K sleep 59 0 0:02:21 0.1% vmtoolsd/1
13426 root 4032K 3220K cpu0 49 0 0:00:00 0.1% prstat/1
5 root 0K 0K sleep 0 -20 0:00:11 0.0% zpool-rpool/13
5 root 0K 0K sleep 0 -20 0:00:10 0.0% zpool-rpool/38
3374 root 61M 15M sleep 59 0 0:00:18 0.0% Xorg/1
3436 root 11M 7532K sleep 59 0 0:00:14 0.0% dtgreet/1
13161 root 3008K 1772K sleep 49 0 0:00:00 0.0% bash/1
Total: 70 processes, 337 lwps, load averages: 0.01, 0.02, 0.01
prstat -mL
This command is very useful to find out where is the actual issue on the system.The blow output shows the percentage of time spent in a given microstate over the last sampling period (default is 5 seconds), beginning with the USR column, up to and including the LAT column. The values in the columns USR through LAT should total 100, accounting for 100% of the threads time for the last sampling period.
# prstat -mL
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
13660 root 11 88 0.0 0.0 0.0 0.0 0.0 0.3 0 227 52K 0 prstat/1
3437 root 5.0 3.3 0.0 0.0 0.0 0.0 89 3.1 5K 24 4K 0 /0
5 root 0.0 2.3 0.0 0.0 0.0 0.0 96 1.5 59 5 0 0 zpool-rpool/12
5 root 0.0 2.3 0.0 0.0 0.0 0.0 96 1.6 59 5 0 0 zpool-rpool/39
5 root 0.0 0.8 0.0 0.0 0.0 0.0 98 1.1 8 1 0 0 zpool-rpool/19
5 root 0.0 0.7 0.0 0.0 0.0 0.0 98 1.1 8 1 0 0 zpool-rpool/22
5 root 0.0 0.7 0.0 0.0 0.0 0.0 98 1.1 8 1 0 0 zpool-rpool/25
| System Bottleneck determination using prstat -mL | |
|---|---|
| Description | Fields |
| Other users and application (Like top command usr field) | USR |
| Kernel (Like top command sys fields) | SYS |
| Monitor the amount of time threads wait for a processor (CPU bottleneck identification) | LAT |
| Memory paging waits (Memory bottleneck identification) | TFL and DTL |
| Disk and Network waits (I/O bottlenect identification ) | SLP |
prstat -s
# prstat -s cpu
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
13150 root 6448K 3744K sleep 39 0 0:00:01 0.5% sshd/1
5 root 0K 0K sleep 99 -20 0:38:09 0.2% zpool-rpool/39
1921 root 6724K 4076K sleep 59 0 0:02:29 0.2% vmtoolsd/1
1611 daemon 4728K 1364K sleep 59 0 0:00:22 0.0% nfsmapid/4
3436 root 11M 7532K sleep 59 0 0:00:16 0.0% dtgreet/1
14506 root 3728K 2924K cpu0 59 0 0:00:00 0.0% prstat/1
# prstat -s pri
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
347 root 0K 0K sleep 99 -20 0:00:00 0.0% zpool-lzpool1/39
351 root 0K 0K sleep 99 -20 0:00:00 0.0% zpool-szpool/39
5 root 0K 0K sleep 99 -20 0:38:10 0.1% zpool-rpool/39
1793 daemon 2468K 1344K sleep 60 -20 0:00:00 0.0% nfsd/2
1620 daemon 2428K 1312K sleep 60 -20 0:00:00 0.0% lockd/2
1610 daemon 2448K 1288K sleep 60 -20 0:00:00 0.0% nfs4cbd/2
1772 root 3260K 2288K sleep 59 0 0:00:01 0.0% vold/6
# prstat -s rss
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
3374 root 61M 15M sleep 58 0 0:00:20 0.0% Xorg/1
1826 root 16M 12M sleep 59 0 0:00:02 0.0% fmd/19
11 root 12M 11M sleep 59 0 0:00:06 0.0% svc.configd/15
9 root 12M 10M sleep 59 0 0:00:02 0.0% svc.startd/13
67 root 12M 9536K sleep 59 0 0:00:02 0.0% vxconfigd/3
3436 root 11M 7532K sleep 59 0 0:00:16 0.0% dtgreet/1
# prstat -s size
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
3374 root 61M 15M sleep 59 0 0:00:20 0.0% Xorg/1
1826 root 16M 12M sleep 59 0 0:00:02 0.0% fmd/19
9 root 12M 10M sleep 59 0 0:00:02 0.0% svc.startd/13
11 root 12M 11M sleep 59 0 0:00:06 0.0% svc.configd/15
67 root 12M 9536K sleep 59 0 0:00:02 0.0% vxconfigd/3
3436 root 11M 7532K sleep 59 0 0:00:16 0.0% dtgreet/1
# prstat -s time
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1921 root 6724K 4080K run 59 0 0:02:29 0.2% vmtoolsd/1
3374 root 61M 15M sleep 59 0 0:00:20 0.0% Xorg/1
3436 root 11M 7532K sleep 59 0 0:00:16 0.0% dtgreet/1
45 root 2456K 1164K sleep 59 0 0:00:06 0.0% dhcpagent/1
3414 root 3968K 1532K sleep 59 0 0:00:02 0.0% veaintf/1
13150 root 6448K 3744K sleep 59 0 0:00:02 0.7% sshd/1
prstat -t
# prstat -t
NPROC USERNAME SWAP RSS MEMORY TIME CPU
62 root 94M 101M 14% 0:41:55 4.9%
8 daemon 8296K 7720K 1.0% 0:00:27 0.0%
prstat -J
# prstat -J
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
13150 root 6448K 3744K sleep 59 0 0:00:02 0.2% sshd/1
1921 root 6724K 4080K sleep 59 0 0:02:30 0.2% vmtoolsd/1
13161 root 3012K 1776K sleep 39 0 0:00:00 0.1% bash/1
5 root 0K 0K sleep 99 -20 0:38:10 0.1% zpool-rpool/39
14651 root 3728K 2924K cpu1 49 0 0:00:00 0.1% prstat/1
PROJID NPROC SWAP RSS MEMORY TIME CPU PROJECT
1 4 3800K 7432K 1.0% 0:00:02 0.4% user.root
0 66 99M 103M 14% 0:42:18 0.3% system
prstat -Z
# prstat -Z
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
5 root 0K 0K sleep 99 -20 0:38:15 8.5% zpool-rpool/39
14781 root 9388K 8152K run 59 0 0:00:03 5.4% svc.configd/11
1826 root 16M 12M sleep 59 0 0:00:08 5.4% fmd/19
14779 root 8440K 7372K run 59 0 0:00:02 2.6% svc.startd/54
349 root 6024K 3176K run 59 0 0:00:03 2.3% devfsadm/8
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
0 71 104M 109M 14% 0:42:40 25% global
1 6 13M 15M 2.0% 0:00:05 8.3% sol1
Hope you have learned many things about prstat command and Solaris process status monitoring
Thank you for reading this article.Please leave a comment if you have any doubt .I will get back to you.
javed says
cpu utilisation in my server shows 100% but i see only 2-3 process that are using 8-9% of CPU. so why rest of 90% cpu is not idle.
sukesh.kota says
what is difference between swap ,RSS ,memory in prstat -a
Lingeswaran R says
RSS- Physical Memory
SWAP – Virtual Memory (Device swap)
tanmay says
Virtual memory = Device swap ?, I think it is both Physical and disk based swap