3

I want to run a multi-parallel task in one bash file like the example code below,

for i in 1 2 3 4 5 6 7 8; do
        setsid python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 &
done
wait  # first wait

echo "next wait"
for i in 9 10 11 12 13 14 15 16; do
        setsid python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 &
done
wait  # second wait

As you can see it, is wait possible to do this? I want to run the first 8 tasks and then wait all the tasks to finish, then spawn the next 8 tasks because the RAM is limited, I cannot run all the 16 tasks in one round.

GoingMyWay
  • 185

3 Answers3

2

Use option -w or --wait to setsid so that the setsid command waits until the python process ends before ending itself. This means the shell wait command now has child processes to wait for.

meuh
  • 6,624
2

Something like this:

set -a pids
for i in {1..8}
do
  python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 &
  pids[${#pids[*]}]=$!  # Remember pid of child
done

# Wait for each child. This loops exits when all children are finished
for pid in ${pids[*]} 
do
  wait $pid
  printf "Sensed end of %d\n" $pid
done

### Continue processing here
xenoid
  • 10,597
1

With GNU Parallel it looks like this:

parallel -j8 setsid python /tmp/t.py {} ::: {1..16} > log
Ole Tange
  • 5,099