5

What's the proper way to nest calls to GNU parallel?

Silly example:

seq 1 100 | parallel echo {} | parallel seq {} 1000

My understanding is that on an 8-CPU box, each parallel would launch 8 jobs for a total of 64 jobs. If you're calling something more substantial than seq this could potentially overload the box. Is there a way to limit the number of jobs but still make full use of parallelism?

Adrian Frühwirth
  • 42,970
  • 10
  • 60
  • 71
Thomas Johnson
  • 10,776
  • 18
  • 60
  • 98
  • The above would start 16 jobs: 8 echo jobs and 8 seq jobs. You probably wanted: `seq 1 100 | parallel "echo {} | parallel -I // seq // 1000"` which would start 64 seqs on an 8 core machine – Ole Tange May 02 '14 at 17:45

1 Answers1

1

Use -j to limit either the outer or the inner parallel:

seq 1 100 | parallel -j1 "echo {} | parallel -I // seq // 1000"

Often you can express the nested command using mulitple :::: or ::: like this:

parallel seq {1} {3} {2} :::: <(seq 10) <(seq 20 30) ::: 1 2 3

It is better because you will keep 8 jobs running at all time where as the above will in periods run fewer than 8 jobs on an 8 core machine.

Ole Tange
  • 31,768
  • 5
  • 86
  • 104