Having a subprocess output redirected to a log file whose name is the pid of the subprocess

Question

Assume that I create a subprocess:

import subprocess
ps = subprocess.Popen(command, shell=True)

I want the output of the new process redirected into a file named x.log, where x is the pid of the created process.

How can this be done?

I try to run multiple processes on a remote server, and each process takes days to complete. I'd like to be able to view the progress of the process which are still running after the script that creates them is done running, so I need their output in files, and the ability to understand which file was generated by which pid.

Currently, I'm drawing a random number r before creating the process, redirecting its output into r.log and maintain a mapping between pids and their output files. This seems stupid, as all I need is to make the process redirect its output to a file whose name is the process pid.

score 2 · Answer 1 · edited May 23 '17 at 12:03

stdout and stderr arguments of subprocess.Popen allow you to redirect standard outputs to different kind of stream.

stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, an existing file descriptor (a positive integer), an existing file object, and None. [...] the child’s file handles will be inherited from the parent. Additionally, stderr can be STDOUT, which indicates that the stderr data from the child process should be captured into the same file handle as for stdout.

You might want to try to redirect the output to file here but you won't have the pid of the child process to properly name the output file.

If your script runs on Unix, you can also try to pass a callable object to Popen's argument preexec_fn which would replace the default output stream with a file handler. With this solution, you can actually get the current process pid with os.getpid.

If preexec_fn is set to a callable object, this object will be called in the child process just before the child is executed. (Unix only)

I did some tests with the preexec_fn solution and just replacing sys.stdout doesn't work. But doing it at the file descriptor level seems to work.

def redirect_output():
    import sys, os
    # The opened file needs to stay alive
    redirect_output._file = open('{}.log'.format(os.getpid()), 'w')

    os.dup2(redirect_output._file.fileno(), sys.stdout.fileno())

sub = subprocess.Popen(['ls'], preexec_fn=redirect_output)

Of course, this is a quick example, some polishing might be needed.

The above example seems to work ok when shell = False but in the case of shell = True, you get the pid of the shell. One way to work around that is to use bash and exec to run the command.

command = 'python -c "import os; print \'\\\'\'process: \'\\\'\', os.getpid()"'
sub = subprocess.Popen(["bash -c 'exec {} > $$.log'".format(command)],
    shell = True,
    preexec_fn = os.setsid
    )
print 'Shell: ', sub.pid

Here os.setsid is used to handle properly termination of the spawned process.

Yes, I know about the `stdout` argument, however, I don't know how to use it for redirecting the output into a file whose name is the process pid... The pid only seems available after the Popen command is executed, at which point I'm not sure how to make the subprocess redirect its output. — R B, Oct 23 '15 at 09:51
@RB if you absolutely need the pid you won't be able to use the ``stdout`` argument indeed. What about the second solution with ``preexec_fn``, could it fit your process? — Gall, Oct 23 '15 at 09:55
You could try to suspend the subprocess when you create it, get its id, redirect its output to the file, and then send a signal from the parent to resume it. I gave it a quick try but could not achieve this... — Emilien, Oct 23 '15 at 10:45
I'm not sure I understand how to use this `preexec_fn`. The child process is a compiled code I can not change, how do I make it check its pid? — R B, Oct 23 '15 at 10:55
This almost works :). It generates it under the pid of the shell which runs the command. — R B, Oct 25 '15 at 11:56
@RB Ho I see, I was missing the ``shell = True`` in my test. You're right, you get the pid of the ``shell`` but it is the same with ``sub.pid``. The only way I can think of right now would be to redirect the output in the subprocess. If it is not an option, well, maybe [process groups](https://docs.python.org/2.7/library/os.html?highlight=os.setsid#process-parameters) could help? — Gall, Oct 26 '15 at 09:20
@RB Ok I found a way that works with ``bash``. It might be tricky depending on your command line I guess... — Gall, Oct 26 '15 at 10:36

score 0 · Answer 2 · answered Oct 23 '15 at 08:08

0

Trying using "currentSubProcess = subprocess.check_output("your Command")" in your python code. Such that "currentSubProcess" will contain output of your subprocess and you can write it in your log file.

answered Oct 23 '15 at 08:08

MhP

56
1

Thanks for the answer, but I'm not sure if this helps. I try to run multiple processes on a remote server, and each process takes days to complete. I'd like to be able to view the progress of the process which are still running after the script that creates them is done running, so I need their output in files. – R B Oct 23 '15 at 08:44

Having a subprocess output redirected to a log file whose name is the pid of the subprocess

2 Answers2