1

I'd like to use qsub to submit a job which runs some Python code which uses Numpy. Numpy is installed in a conda virtual environment which I activate upon logging into the cluster, and I can import it if I simply call import numpy in the Python command line interpreter.

However, when I submit a job with qsub, it tries to run the job without using that environment. After some testing it seems that I can get qsub to run things in the right environment if I pass in all variables with the -V option.

However, this muddles up other parts of the scripts. It would be much nicer if I could pass in only what I need to get qsub to run in the right environment. How can I do this?

P.S. The solution presented here did not work for me; the error was Unable to locate a modulefile for 'numpy'. I'd assume it's not in the current path but I'm not sure where it.

BGreen
  • 370
  • 3
  • 17
  • You can run scripts in a specific environment with `conda run -n my_env python my_script.py`. Is that what you need? – merv Feb 07 '21 at 07:17
  • Thank you for the suggestion! At the time, the command `conda` wasn't recognized because it wasn't in the path. This has been fixed with the first line in the solution below. Please excuse me for not remembering to follow up with it earlier. I wouldn't be surprised if your solution also worked. – BGreen Feb 07 '21 at 14:42
  • This probably could have also been solved by using a `#!/bin/bash -l` shebang in the script. – merv Feb 15 '21 at 22:35

1 Answers1

1

I've got a solution now. In the script that is submitted with qsub I've added two lines to have it reload the environment.

source [Intel Parallel Studios script that sets up conda environment variables]
source activate my_root
BGreen
  • 370
  • 3
  • 17