10

Is it possible to expand variables in comments inside a bash script?

I want to write a script to feed into SGE. The qsub syntax allows me to pass additional parameters to the grid engine using lines inside the bash script which begin with #$. For example,

#$ -q all.q
#$ -S /bin/bash
#$ -V
#$ -m beas
#$ -o run_20120103.out

What I want is that the -o parameter is dynamically set to a variable, say $1. So I would naively write

#$ -o run_${1}.out

However, since the line starts with a #, bash ignores it and the variable $1 is not expanded.

Any ideas? Some bash preprocessor? Some other way?

EDIT I just chose $1 as an example. It could just as well be $FOO or $BAR.

andreas-h
  • 10,679
  • 18
  • 60
  • 78
  • What is determining the value of ${1} ? Are you doing a for cycle to load different jobs on the cluster/grid? – lev.tuby Feb 05 '13 at 18:25
  • 2
    In that case you can simply call qsub with the option -o inside your loop instead of putting #$ -o inside the script fed into qsub. – lev.tuby Feb 05 '13 at 23:00

3 Answers3

5

Variable expansion happens in shell memory, it doesn't affect the file. Therefore, it doesn't matter what bash expands.

Instead, you can probably generate the script to be run on the fly, with everything expanded in place:

cat << EOF | qsub [options] -
#$ -o run_$1.out
cmds
EOF
that other guy
  • 116,971
  • 11
  • 170
  • 194
  • Can you explain exactly what this script does step by step? I tried testing it but as I have no idea what the behavior should be and I am not getting the behavior I expect it's quite useless – Gullydwarf Apr 02 '15 at 10:32
  • @Gullydwarf This script is not a complete or self contained example. It merely illustrates a syntactic structure for expanding variables in text before feeding it to qsub. – that other guy Apr 02 '15 at 14:52
  • Could you expand it to be? I think this might be what I am looking for to expand a variable walltime for my jobs. There is no option like -o for walltime in qsub is there? That would make it so much easier :p Edit: I think I found it, -l can also be used as option to qsub, where I think I can pass the walltime. Testing now – Gullydwarf Apr 07 '15 at 13:13
  • Unfortunately I only know bash, not qsub. If you know how to invoke qsub and what the input should be, I'll be happy to help out with the bash required to make that happen. – that other guy Apr 07 '15 at 14:24
  • The input for qsub is a bash job script :) – Gullydwarf Apr 09 '15 at 13:49
  • Well, if you know how to invoke qsub and what that bash job script should look like... – that other guy Apr 09 '15 at 14:33
0

To be a bit more SGE specific than the accepted answer, just redirect the standard output of qsub and parse the job id. You can then use this to alter the job name via qalter.

Here's a sample of a script I used to submit and modify jobs like this.

 for a in $args;
 do 
    qsub submission_script $a 1> job.txt
    jid=$(cat job.txt | tr -dc '0-9' | cut -c -6)
    qalter $jid -N $a
 done

I'm certain that there are more idiomatic ways to do this, but this is a first go.

This approach allows submission_script to have it's own arguments. It also allows you to programmatically alter other job characteristics, not just the name.

patr1ckm
  • 1,176
  • 9
  • 12
-2

I am also using the qsub command in SGE to generate different output or error logs. For example, if I would like to include the TASK_ID in the script, instead of doing

#$ -o run_${TASK_ID}.out

Write it without the paretheses will do the trick:

#$ -o run_$TASK_ID.out
En-Chuan Huang
  • 11
  • 1
  • No, it doesn't do the trick. You get a file called run_$TASK_ID.out, the variable hasn't be replaced by its value – matthieu Feb 20 '17 at 14:20