Start background process in python via subprocess and write output to file

Question

Dear stackoverflow users,

I'm looking for a solution for a probably quite easy problem. I want to automate some quantum chemical calculations and ran into a small problem.

Normally you start your quantum chemical programm (in my case it's called orca) with your input file (*.inp) on a remote server as a background process and pipe the output into an outputfile (*.out) via

nohup orca H2.inp >& H2.out &

or something similar.

Now I wanted to use a python script (with some templating) to write the input file automatically. At the end the script should start the calculation in a way that I could log out of the server without stopping orca. I tried that with

subprocess.run(["orca", input_file], stdout=output_file)

but so far it did not work. How do I "emulate" the command given at the top with the subprocess module?

Regards

Update I have one file that is called H2.xyz. The script reads and splits the filename by the point and creates an input file name H2.inp and the output should be written into the file H2.out.

Update 2 The input file is derived from the *xyz file via

xyzfile = str(sys.argv[1])
input_file = xyzfile.split(".")[0] + ".inp"
output_file = xyzfile.split(".")[0] + ".out"

and is created within the script via templating. In the end I want to run the script in the following way:

python3 script.py H2_0_1.xyz

"If the shell is invoked explicitly, via shell=True, it is the application’s responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid shell injection vulnerabilities" from https://docs.python.org/3/library/subprocess.html#security-considerations — Sharku, Aug 30 '18 at 11:35
is `output_file` the path to a file (I don't think that will work) or a file-like object (like what you get from using the `open(output_file, 'w')` method? — Daren Thomas, Aug 30 '18 at 12:01
So far I did not get it to work, so I'm open to any solution. — JakobJakobson13, Aug 30 '18 at 13:40

Adelina · Answer 1 · 2018-08-30T11:58:11.363

1

Why not simply:

subprocess.Popen(f'orca {input_file} >& {output_file}',
                 shell=True, stdin=None, stdout=None, stderr=None, close_fds=True)

More info: Run Process and Don't Wait

edited Aug 30 '18 at 11:58

answered Aug 30 '18 at 11:52

Adelina

10,915
1
38
46

like I commented in the question shell=True is not secure, because of injection vulnerabilities. In my solution Python is taking care of that. But yes I might be over killing it xDDD – Sharku Aug 30 '18 at 11:54

score 1 · Answer 2 · answered Aug 30 '18 at 13:52

For me (Windows, Python 2.7) the method call works very fine like this:

with open('H2.out', 'a') as out :
    subprocess.call(['orca', infile], stdout=out,
                                      stderr=out,
                                      shell=True)   # Yes, I know. But It's Windows.

On Linux you maybe do not need shell=True for a list of arguments.

Sharku · Answer 3 · 2018-08-30T13:00:27.687

I had the same problem not long ago. Here is my solution:

commandLineCode = "nohup orca H2.inp >& H2.out &"
try:
    proc = subprocess.Popen(commandLineCode,
                            stdin=subprocess.PIPE,
                            stdout=subprocess.PIPE,
                            stderr=subprocess.PIPE,
                            cwd = workingDir)

except OSError:
    print("Windows Error occured")
    print(traceback.format_exc())

timeoutInSeconds = 100
try:
    outs, errs = proc.communicate(timeout = timeoutInSeconds)
except subprocess.TimeoutExpired:
    print("timeout")
    proc.kill()
    outs, errs = proc.communicate()

stdoutDecode = outs.decode("utf-8")
stderrDecode = errs.decode("utf-8")

for line in stdoutDecode.splitlines():
    # write line to outputFile
if stderrDecode:
    for line in stderrDecode.splitlines():
        # write line to error log

The OSError exception is pretty important since you never now what your OS might do wrong.

For more on the communicate() command which actually starts the process read: https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate

score 0 · Answer 4 · answered Aug 30 '18 at 12:06

0

Is the usage of subprocess important? If not, you could use os.system.

The Python call would get really short, in your case

os.system("nohup orca H2.inp >& H2.out &")

should do the trick.

answered Aug 30 '18 at 12:06

Simon K.

56
5

Start background process in python via subprocess and write output to file

4 Answers4