4

for my master's thesis I'm trying to get resource usage information about MQTT brokers on a debian machine.

I'm having troubles finding a way to find out how much total cpu time debian has spent on a process I'm doing resource usage statistics for. I've tried out pidstat and cpustat to get cpu statistics about this process, but either:

  • the cpu time is shown as a percentage (pidstat)
  • minimal measuring interval of 1s is too small for short-lived processes(cpustat).

Can anyone try to lead me to a way I can find out how much total CPU time (or even CPU cycles) a process has used?

Justin Liu
  • 139
Melvin Bender
  • 41

2 Answers2

3

atop is a great tool that should work for your case. It is an improved top tool with accounting (which is exactly what you need for the reasons I'll write out next).

atop is a daemon that accounts for a lot of statistics per X seconds (however you configure it, default on Debian is 60s) and it saves a history of all processes and their usages on the system, basically like top snapshots.

atop also has the functionality to print out specific statistics in a parseable format:

atop -r /var/log/atop/atop_<date> -P <format>

This will read the process history from a log file from the specified date and print it out in a machine parseable format specific for certain statistics.

The format you need is PRC:

PRC  Process and thread level totals.
            This  line  contains the total cpu time consumed in system mode (`sys') and in user mode (`user'), the total number of processes present at this moment (`#proc'), the total number of threads present at this moment in state `running' (`#trun'), `sleeping
            interruptible' (`#tslpi') and `sleeping uninterruptible' (`#tslpu'), the number of zombie processes (`#zombie'), the number of clone system calls (`clones'), and the number of processes that ended during the interval (`#exit') when process accounting is
            used. Instead of `#exit` the last column may indicate that process accounting could not be activated (`no procacct`).
            If the screen-width does not allow all of these counters, only a relevant subset is shown.

For example, with the following command you will get an output like the following:

$ atop -r /var/log/atop/atop_20200617 -P PRC
PRC hostname 1592403710 2020/06/17 14:21:50 5000098 377 (google_osconfig) S 100 262341 41122 0 120 0 0 0 0 377 y

Where the total SYS CPU time is 262341 and total USR CPU time is 41122 for the process google_osconfig with PID 377.

atopacct.service is the service that does the accounting and it is a deamon, so it accounts even for short-running processes.

You can install it on Debian 9 via the package manager:

sudo apt-get update
sudo apt-get install atop

And it will automatically start accounting. You can read more in the manpage.

Fanatique
  • 5,153
2

Use time(1). It may not be accurate enough for really short-lived processes though.

Keep in mind the time shell builtin is not the time standalone executable. In Bash help time describes the builtin; man 1 time describes the executable.

If you can run the tool you want to measure by yourself (i.e. you choose the command to run) then it's quite simple. Instead of the-tool argument1 argument2 run

time the-tool argument1 argument2
# or
/usr/bin/time the-tool argument1 argument2

If the-tool is run by some other tool and you cannot easily make it run time the-tool … instead, then create a wrapper script:

  1. Move the-tool to another name, e.g. the-real-tool.

  2. Create a script named the-tool, make it executable and reachable via $PATH. The content:

    #!/bin/sh
exec /usr/bin/time the-real-tool "$@"

(Alternatively you may not rename the-tool, but only place the script as the-tool in a directory that appears early in $PATH, so the other tool will find the script when it tries to run the-tool. Inside the script you need a full path to the real the-tool, so the script doesn't run itself recursively.)

Study man 1 time and notice --format and --output options. You may want to use -a --output especially in the wrapper script to collect results automatically:

#!/bin/sh
log="/tmp/the-tool-$(date --rfc-3339=seconds)-$$.log"
printf '<%s> ' "$0" "$@" >"$log"
printf '\n' >>"$log"
exec /usr/bin/time -a --output="$log" the-real-tool "$@"

If you use bash and its builtin then you can measure with greater precision. Logging output only from time is somewhat tricky in this case, yet possible:

#!/bin/bash
log="/tmp/the-tool-$(date --rfc-3339=seconds)-$$.log"
printf '<%s> ' "$0" "$@" >"$log"
printf '\n' >>"$log"
{ time the-real-tool "$@" 2>&3; } 3>&2 2>>"$log"

The overhead of time is researched in this answer of mine. It should be negligible in comparison to the resolution of time, especially for the builtin. In case of a wrapper script the interpreter (sh or bash) will add to the total time as well, but not to the results reported by time. In other words the results from measuring the-real-tool should not be inflated but you'll need to wait little longer for the-tool-script than you would wait for the-tool-original (a.k.a. the-real-tool now).

Kamil Maciorowski
  • 81,893