Using a loop variable in a Bash script to pass different command-line arguments

Question

I have a C++ program to which I pass two doubles as inputs from the command line using

int main(int argc, char *argv[]){
    double a,b;
    a = atof(argv[1]);
    b = atof(argv[2]);
    further code.....

I run the code on a cluster using the qsub utility and I have a Bash script named 'jobsub.sh` to submit the jobs which looks like this:

#!/bin/csh -f
hostname
cd /home/roy/codes/3D             # Change directory first -- replace Mysubdir
set startdir = `pwd`               # Remember the directory we're in
if( ! -d /scratch/$USER ) then
    mkdir /scratch/$USER       # Create scratch directory
endif                              # If it does not exist
#cp infile12 /scratch/$USER     # Copy input file to scratch directory
cd /scratch/$USER                  # Change to scratch directory
#rm *.*
$HOME/codes/3D/autoA100.out 2.1 2.2          # Run a program
cp * $startdir         # Copy outputfiles back to where we started

At the terminal I do qsub jobsub.sh.

However, I want to run the same executable for different values of a and b in parallel on different cores. Is it possible to write a for loop in the Bash script so that I can do something like,

for i=1;i<=10;i++ {
   $HOME/codes/3D/autoA100.out 2+i*0.1 2+i*0.2
}

Your "bash" script resembles bash, but contains many syntax errors. — jordanm, Jul 06 '12 at 01:21
@jordanm: In some ways it looks like csh (`set` for example). — Dennis Williamson, Jul 06 '12 at 02:05
@DennisWilliamson I edited the script in the post, it actually is a csh — lovespeed, Jul 06 '12 at 12:13
Your question is tagged [tag:bash], you refer to "bash" in the text and your script is named `jobsub.sh` (but it's actually csh). Do you want answers in Bash or Csh? By the way, [Csh Programming Considered Harmful](http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/). — Dennis Williamson, Jul 06 '12 at 12:29
I am sorry to say I am not really aware of the difference between csh and bash, I inherited this script from someone who called it a bash script. @DennisWilliamson — lovespeed, Jul 06 '12 at 15:41
They are two completely different shells with very different syntaxes. They share some common features, though, as most Unix shells must do. — Dennis Williamson, Jul 06 '12 at 15:48

score 3 · Answer 1 · answered Jul 06 '12 at 00:26

If you are submitting the execution script to a batch loader, then there is no way to have a simple loop do the execution like you want because the entire script is run on each node. However, most batch loaders provide environment variables to the user.

PBS, for example, has $PBS_ARRAYID, which specifies the unique ID for the job as it is running. So instead of using a loop, your script can have:

a=$(echo "2+($PBS_ARRAYID+1)*0.1" | bc -l)
b=$(echo "2+($PBS_ARRAYID+1)*0.2" | bc -l)
$HOME/codes/3D/autoA100.out $a $b

Notice I've added 1 to $PBS_ARRAYID above because the ID begins from 0 while your loop begins from 1. It's also worth mentioning that while bash can do some arithmetic natively, it cannot handle real numbers; that's why I had to invoke bc.

It can handle some real numbers, integers. It can not do floating point math. — jordanm, Jul 06 '12 at 01:30
@chrisaycock sorry to revive this old post but there has never been an accepted solution; you said " there is no way to have a simple loop do the execution like you want because the entire script is run on each node." How is this solved using two scripts, one for the loop and one for qsub? — forest.peterson, Aug 25 '13 at 05:29

Jo So · Answer 2 · 2012-07-06T01:39:49.570

0

You can always run jobs in the background (in which case the shell will continue with the next instruction)

MY_PIDS=
for i in 1 2 3 4 5 6 7 8 9 10
do
    yourjob "$(a i)" "$(b i)" &   # Note the ampersand (background)
    MY_PIDS="$MY_PIDS "$!         # Do not write the $! inside the quotes, bash will
                                  # break (better, don't use bash but plain sh)
done

# Wait wor each job to complete and cleanup
for p in $MY_PIDS
do
    wait "$p"
    echo "Return code of job $p is $?"
    do_cleanup_for "$p"
done

But certainly you need to make sure that the jobs running in parallel are not stepping on each other's feet (like writing to the same file).

edited Jul 06 '12 at 01:39

answered Jul 06 '12 at 00:26

Jo So

25,005
6
42
59

Quoting $! is only a problem in interactive shells. – jordanm Jul 06 '12 at 01:35
Enough of a problem to not quote it above, I'd say. – Jo So Jul 06 '12 at 01:37
Is there a reason you used a string to store pids instead of an array? – jordanm Jul 06 '12 at 01:43
Bro, the OP tagged the question [tag:bash]. – Dennis Williamson Jul 06 '12 at 02:03
What's wrong? Works in bash, too. Easy to understand too, though probably not the answer to OP's question ;-) – Jo So Jul 06 '12 at 13:50

Using a loop variable in a Bash script to pass different command-line arguments

2 Answers2

Linked