Welcome, Guest. [ Log In ]
Question   Process Accounting (to find and remove the cause of slowness)
Search KBase


Top 3 in this Area:
1. MySQL Optimization: Slow Query Logs and Indexes
2. Process Accounting (to find and remove the cause of slowness)
3. General MySQL Optimization Tips

 
Process Accounting (to find and remove the cause of slowness)
Unix Process Accounting is a tool that literally takes account of every single process on your server, recording how much CPU (processor) and memory (RAM) each process consumes. When summarized, this information can be very useful for tracking down overzealous processes and hogs of resources so that you may restore your server to it's rightful speed and performance!

DreamHost keeps a daily accounting of your processes, which can be found in /var/log/sa/ (note: you may have to request DreamHost to provide you with access to those files). For example, /var/log/sa/sa.itemized.0 is an itemized report of all the processes run from yesterday, sa.itemized.1 is the itemized report for the day before yesterday, sa.itemized.2 is the itemized report for two days before yesterday. Along those same lines is sa.summary.0, which is a summary of each user's consumption of resources.

What does all of this mean? The best way to use this information is to examine the CPU-minutes. A CPU-minute can be considered a unit of work for a server. No, CPU-minutes are not "real time", that is, there are not 60minutesX24hours=1440 CPU-minutes in a day. If a machine is idle, doing nothing at all, that doesn't count towards CPU time. Only when the processor is actually doing something does it count as CPU-minutes.

So knowing how many CPU-minutes per day your users and processes are consuming can help you to find the user or process which is causing problems for the entire machine. The best way to understand all of this is to breeze through the examples below, and then try it out on your own machine.

One final confusing note: Yes, there are CPU-seconds as well:
1 CPU-minute = 60 CPU-seconds

In the examples below:
"cp" = CPU-minute
"cpu" = CPU-second

# Give me a summary of the 10 "biggest" users
# (Note that the entire machine utilized 599 CPU-minutes for the entire day yesterday)
yourserver:# head /var/log/sa/sa.summary.0 
                                   133573  100.00% 46089720.15re  100.00%     599.89cp  100.00%         0avio  78478595k*sec
bob                                 84064   62.93%  618688.32re    1.34%     349.68cp   58.29%         0avio  35499807k*sec
mysql                                2158    1.62% 29884017.65re   64.84%     147.47cp   24.58%         0avio 413986887k*sec
dhapache                             9201    6.89% 14111172.13re   30.62%      87.92cp   14.66%         0avio  17167069k*sec
root                                35136   26.30%  860049.95re    1.87%      14.14cp    2.36%         0avio    395749k*sec
postfix                              1267    0.95%  612272.05re    1.33%       0.44cp    0.07%         0avio     21756k*sec
mailuser1                             271    0.20%     542.82re    0.00%       0.03cp    0.01%         0avio       728k*sec
mailuser2                             271    0.20%     612.53re    0.00%       0.03cp    0.01%         0avio       673k*sec
louanne                               271    0.20%     539.83re    0.00%       0.03cp    0.01%         0avio       659k*sec
sarahlee                              271    0.20%     610.13re    0.00%       0.03cp    0.01%         0avio       655k*sec

# Give me an analysis of all the processes run by all the users.
yourserver:# /dh/misc/bin/sa.analysis.pl /var/log/sa/sa.itemized.0 | head
Process               CPU seconds      user   machine   count  average
php.cgi                20980.6200   58.291%   87.419%   84063    0.250
mysqld                  8786.3600   24.411%   36.610%    2217    3.963
libhttpd.ep             5290.4900   14.699%   22.044%    8650    0.612
ps                       286.3300    0.796%    1.193%     726    0.394
sshd                     266.5500    0.741%    1.111%    1459    0.183
analog                    83.8400    0.233%    0.349%     204    0.411
mysqld                    62.3700    0.173%    0.260%      12    5.197
analyze_log.pl            37.9200    0.105%    0.158%       5    7.584
top                       26.7400    0.074%    0.111%       2   13.370
gzip                      18.5000    0.051%    0.077%      85    0.218
# That was the information that you really want to look at.
# Notice that for this machine, php.cgi was run a whopping 
# 84,000 times in one day (about once a second), consuming
# about 350 CPU-minutes that day (which is a lot!).  But it only
# consumed on average 0.25 CPU-minutes.  It looks pretty efficient,
# yet even a 15% increase in efficiency would drop the amount of 
# resources consumed by php.cgi to under 300.

# Give me a summary of all the processes run by the user `bob' yesterday:
yourserver:# grep ^bob[[:blank:]] /var/log/sa/sa.itemized.0 > /tmp/asdf
yourserver:# /dh/misc/bin/sa.analysis.pl /tmp/asdf |head -10
Process               CPU seconds      user   machine   count  average
php.cgi                20980.6200   99.999%   87.419%   84063    0.250
dothis.pl              0.1300    0.001%    0.001%       1    0.130

# Give me some of the raw data for the fun of it
yourserver:# tail -10 /var/log/sa/sa.itemized.0
root       0.01 cpu      311k mem      0 io chmod 
dhapache   0.04 cpu     3370k mem      0 io libhttpd.ep      *
root       0.00 cpu      314k mem      0 io date   
root       0.02 cpu      514k mem      0 io savelog
bob        0.23 cpu     1723k mem      0 io php.cgi
bob        0.11 cpu     1723k mem      0 io php.cgi
dhapache   0.01 cpu     3370k mem      0 io libhttpd.ep      *
bob        0.03 cpu     1719k mem      0 io php.cgi          
root       0.01 cpu      310k mem      0 io uname
dhapache   0.02 cpu     3370k mem      0 io libhttpd.ep      *
root       0.00 cpu      310k mem      0 io cut

Last updated: Jun 19, 2003.