Monitor the CPU usage of an OpenFOAM simulation running on a slurm job

Question

I'm running an OpenFOAM simulation on a cluster. I have used the Scotch decomposition method and my decomposeParDict looks like this:

FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}
numberOfSubdomains 6;
method          scotch;

checkMesh and decomposePar finish with no issues. I have assigned 6 nodes to the slurm by

srun -N6 -l sonicFoam

and the solver runs smoothly without any errors.

The issue is the solution speed is not improved in comparison to the non-parallel simulation I ran before. I want to monitor the CPU usage to see if all of the 6 nodes I have assigned are similarly loaded. The squeue --user=foobar command return the jobNumber and list of nodes assigned (NODELIST(REASON)) which looks like this:

foo,bar[061-065]

from sinfo command these nodes are both in debug and main* PARTITIONs (which I have absolutely no idea what it means!).

This post says that you can use the sacct or sstat commands to monitor CPU time and memory usage of a slurm job. But when I run

sacct --format="CPUTime,MaxRSS"

it gives me:

 CPUTime     MaxRSS
---------- ----------
  00:00:00
  00:00:00
  00:07:36
  00:00:56
  00:00:26
  15:26:24

which I can not understand. And when I specify the job number by

sacct --job=<jobNumber> --format="UserCPU"

The return is empty. So my questions are

• Is my simulation loading all nodes or is it running on one or two and the rest are free?
• am I running the right commands? if yes what those numbers mean? how they represent the CPU usage per node?
• If not then what are the right --format="..."s for sacct and/or sstat (or maybe other slurm commands) to get the CPU usage/load?

P.S.1. I have followed the OpenFOAM compiling following the official instructions. I did not do anything with OpenMPI and it's mpicc compiler for that matter though.

P.S.2 For those of you who might end up here. Maybe I'm running the wrong command apparently one can first allocate some resources by:

srun -N 1 --ntasks-per-node=7 --pty bash

where 7 is the number of cores you want and bash is just a name. and then run the solver with:

mpirun -np 7 sonicFoam -parallel -fileHandler uncollated

I'm not sure yet though.

Answer 1

You can use

sacct --format='jobid,AveCPU,MinCPU,MinCPUTask,MinCPUNode'

to check whether all CPUs have been active. Compare AveCPU (average CPU time of all tasks in job) with MinCPU (minimum CPU time of all tasks in job). If they are equal, all 6 tasks (you requested 6 nodes, with, implicitly, 1 task per node) worked equally. If they are not equal, or even MinCPU is zero, then some tasks have been doing nothing.

But in your case, I believe you will observe that all tasks have been working hard, but they were all doing the same thing.

Besides the remark concerning the -parallel flag by @timdykes, you also must be aware that launching an MPI job with sun requires that OpenMPI was compiled with Slurm support. During your installation of OpenFOAM, it installed its own version of OpenMPI, and if file /usr/include/slurm/slurm.h or /usr/include/slurm.h exists, then Slurm support was probably compiled in. But the safest is probably to use mpirun.

But to do that, you will have to first request an allocation from Slurm with either sbatch or salloc.

Answer 2

Have you tried running with the '-parallel' argument? All of the OpenFOAM examples online use this argument when running a parallel job, one example is the official guide for running in parallel.

srun -N $NTASKS -l sonicFOAM -parallel

As an aside - I saw you built openfoam yourself, have you checked whether the cluster admins have provided a module for it? You can usually run module avail to see a list of the available modules, and then module load moduleName if there is an existing OpenFOAM module. This is useful as you can probably trust its been built with all the right options and would automatically set up your $PATH etc.