How to find from where a job is submitted in SLURM?

Question

I submitted several jobs via SLURM to our school's HPC cluster. Because the shell scripts all have the same name, so the job names appear exactly the same. It looks like

[myUserName@rclogin06 ~]$ sacct -u myUserName
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
12577766         run.sh    general ourQueue_+          4    RUNNING      0:0 
12659777         run.sh    general ourQueue_+          8    RUNNING      0:0 
12675983         run.sh    general ourQueue_+         16    RUNNING      0:0

How can I know from which directory a job is submitted so that I can differentiate the jobs?

score 36 · Accepted Answer · answered Jul 07 '14 at 02:48

36

You can use the scontrol command to see the job details. $ scontrol show job <jobid>

For example, for a running job on our SLURM cluster:

$ scontrol show job 1665191
    JobId=1665191 Name=tasktest
    ...
    Shared=OK Contiguous=0 Licenses=(null) Network=(null)
    Command=/lustre/work/.../slurm_test/task.submit
    WorkDir=/lustre/work/.../slurm_test

You are looking for the last line, WorkDir.

answered Jul 07 '14 at 02:48

Derek

741
5
5

1

If you need the output log file, you can `scontrol show job | grep StdOut`. – Yamaneko Dec 14 '18 at 16:25
after you found the workdir, search for the job name or part of the job name in that dir. If it is sbatch file that has jobname in there you likely can find it quickly this way – Moe Tsao Jun 08 '22 at 18:40

score 13 · Answer 2 · answered Jul 31 '14 at 15:18

13

The latest version of Slurm now offers that information through squeue with :

squeue --format "%Z"

that displays, according to the man page,

%Z    The job’s working directory.

answered Jul 31 '14 at 15:18

damienfrancois

52,978
9
96
110

score 4 · Answer 3 · answered Aug 12 '20 at 15:16

4

In order to list the work directory of past jobs that are no longer accessible via squeue or scontrol, you can use sacct:

sacct -S 2020-08-10 -u myUserName --format "jobid,jobname%20,workdir%70"

Lists job id, job name and work directory of all jobs of user myUserName since August 10th, 2020.

answered Aug 12 '20 at 15:16

leopold.talirz

640
7
19

I'm not seeing the `workdir` field as an option with `sacct 2.6` on `CentOS release 6.5 (Final)` – Leo Sep 10 '22 at 17:24
1

@Leo Slurm 2.6 was released in 2013. Looks like the feature was [added in slurm 17.11.0rc1](https://github.com/SchedMD/slurm/blob/67098bc676a7911968654b7432dc22e50eae924e/NEWS#L4047) – leopold.talirz Sep 11 '22 at 19:08

How to find from where a job is submitted in SLURM?

3 Answers3

Linked