43

The terminology used in the sbatch man page might be a bit confusing. Thus, I want to be sure I am getting the options set right. Suppose I have a task to run on a single node with N threads. Am I correct to assume that I would use --nodes=1 and --ntasks=N?

I am used to thinking about using, for example, pthreads to create N threads within a single process. Is the result of that what they refer to as "cores" or "cpus per task"? CPUs and threads are not the same things in my mind.

Tung
  • 26,371
  • 7
  • 91
  • 115
Tanash
  • 461
  • 1
  • 5
  • 8
  • What would happen if --cpus-per-task exceed the #CPUs per node – vbenara Nov 13 '19 at 03:05
  • I suppose nothing @V.ben as if you set `OMP_NUM_THREADS` to a number greater than the number of cores on your machine. They will simply be "overloaded", but any additional clarification would really be appreciated! – andreagalle Jul 01 '20 at 23:40

1 Answers1

80

Depending on the parallelism you are using: distributed or shared memory

--ntasks=# : Number of "tasks" (use with distributed parallelism).

--ntasks-per-node=# : Number of "tasks" per node (use with distributed parallelism).

--cpus-per-task=# : Number of CPUs allocated to each task (use with shared memory parallelism).

From this question: if every node has 24 cores, is there any difference between these commands?

sbatch --ntasks 24 [...]
sbatch --ntasks 1 --cpus-per-task 24 [...]

Answer: (by Matthew Mjelde)

Yes there is a difference between those two submissions. You are correct that usually ntasks is for mpi and cpus-per-task is for multithreading, but let’s look at your commands:

For your first example, the sbatch --ntasks 24 […] will allocate a job with 24 tasks. These tasks in this case are only 1 CPUs, but may be split across multiple nodes. So you get a total of 24 CPUs across multiple nodes.

For your second example, the sbatch --ntasks 1 --cpus-per-task 24 [...] will allocate a job with 1 task and 24 CPUs for that task. Thus you will get a total of 24 CPUs on a single node.

In other words, a task cannot be split across multiple nodes. Therefore, using --cpus-per-task will ensure it gets allocated to the same node, while using --ntasks can and may allocate it to multiple nodes.

Another good Q&A from CÉCI's support website: Suppose you need 16 cores. Here are some use cases:

  • you use mpi and do not care about where those cores are distributed: --ntasks=16
  • you want to launch 16 independent processes (no communication): --ntasks=16
  • you want those cores to spread across distinct nodes: --ntasks=16 and --ntasks-per-node=1 or --ntasks=16 and --nodes=16
  • you want those cores to spread across distinct nodes and no interference from other jobs: --ntasks=16 --nodes=16 --exclusive
  • you want 16 processes to spread across 8 nodes to have two processes per node: --ntasks=16 --ntasks-per-node=2
  • you want 16 processes to stay on the same node: --ntasks=16 --ntasks-per-node=16
  • you want one process that can use 16 cores for multithreading: --ntasks=1 --cpus-per-task=16
  • you want 4 processes that can use 4 cores each for multithreading: --ntasks=4 --cpus-per-task=4
Community
  • 1
  • 1
Tung
  • 26,371
  • 7
  • 91
  • 115
  • 2
    Thank you for your reply, but in your example `sbatch --ntasks 24 […]`, what if I choose '--nodes=1' ? is it in this case will be same as your second example `sbatch --ntasks 1 --cpus-per-task 24 [...]` which will get a total of 24 CPUs on a single node? – Tanash Jul 02 '18 at 18:50
  • 2
    For your question: you will need `--nodes=1 --ntasks=1 --cpus-per-task=N` where N is the # of (OpenMP) parallel threads. See example here https://stackoverflow.com/a/50654725/786542 – Tung Jul 02 '18 at 18:55
  • Is `sbatch --ntasks 1` indicates to the number of threads in the sbatch script? – Tanash Jul 02 '18 at 19:01
  • 1
    No. The `--cpus-per-task=N` is – Tung Jul 02 '18 at 19:02
  • Does your last comment (about OpenMP) apply when no OpenMP is in use, just pthreads? – Tanash Jul 02 '18 at 19:40