Launch a completely independent process

Question

I want to initiate a process from my python script main.py. Specifically, I want to run the below command:

`nohup python ./myfile.py &`

and the file myfile.py should continue running, even after the main.py script exits.

I also wish to get the pid of the new process.

I tried:

os.spawnl*
os.exec*
subprocess.Popen

and all are terminating the myfile.py when the main.py script exits.

Update: Can I use os.startfile with xdg-open? Is it the right approach?

Example

a = subprocess.Popen([sys.executable, "nohup /usr/bin/python25 /long_process.py &"],\
     stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
print a.pid

If I check ps aux | grep long_process, there is no process running.

long_process.py which keeps on printing some text: no exit.

Am I doing anything wrong here?

Can you post a minimal example that fails to work? After a trivial `python -c 'import subprocess; subprocess.Popen(["sleep", "60"])'` the output of `ps` shows that the `sleep` keeps running just fine after Python has exited. — user4815162342, Dec 23 '14 at 17:15
You are running Python with a single argument, `"nohup /usr/bin/python25 ..."`, which cannot work because the `python` executable will look for a script in a file named exactly `nohup /usr/bin/...`, which doesn't exist. And since you specify `stderr` as `PIPE` without ever reading the pipe's content, you never get to see the error message. Lose the `nohup` and `&`, and simply run `subprocess.Popen([sys.executable, "/.../long_process.py"])`. Also, don't specify `stdin` and `stderr` as pipes unless you mean it. — user4815162342, Dec 23 '14 at 17:21
`nohup` makes most sense when you start a process from a shell. I don't see `shell=True` in your `Popen` call. — 9000, Dec 23 '14 at 17:42

9000 · Accepted Answer · 2014-12-24T16:15:57.723

21

You open your long-running process and keep a pipe to it. So you expect to talk to it. When yor launcher script exits, you can no longer talk to it. ~~The long-running process receives a SIGPIPE and exits.~~

The following just worked for me (Linux, Python 2.7).

Create a long-running executable:

$ echo "sleep 100" > ~/tmp/sleeper.sh

Run Python REPL:

$ python
>>>

import subprocess
import os
p = subprocess.Popen(['/bin/sh', os.path.expanduser('~/tmp/sleeper.sh')])
# look ma, no pipes!
print p.pid
# prints 29893

Exit the REPL and see the process still running:

>>> ^D
$ ps ax | grep sleeper
29893 pts/0    S      0:00 /bin/sh .../tmp/sleeper.sh
29917 pts/0    S+     0:00 grep --color=auto sleeper

If you want to first communicate to the started process and then leave it alone to run further, you have a few options:

Handle SIGPIPE in your long-running process, do not die on it. Live without stdin after the launcher process exits.
Pass whatever you wanted using arguments, environment, or a temporary file.
If you want bidirectional communication, consider using a named pipe (man mkfifo) or a socket, or writing a proper server.
Make the long-running process fork after the initial bi-direcional communication phase is done.

edited Dec 24 '14 at 16:15

answered Dec 23 '14 at 17:34

9000

39,899
9
66
104

Great answer, it's a good point that the process dies due to `SIGPIPE`. (In the code from the question the process never got started, but the OP might have also tried other variants that did start and died due to `SIGPIPE`.) – user4815162342 Dec 23 '14 at 19:50
1

It does not create "a completely independent process" (that what `python-daemon` package does). Also, [a child process won't get SIGPIPE in Python 2](http://stackoverflow.com/a/22083141/4279). Though in simple cases, your solution is enough (you should redirect to `os.devnull` child's stdin/stdout/stderr to avoid waiting for input and/or a spurious output to the terminal). – jfs Dec 23 '14 at 20:35
1

@user4815162342: It won't die due to SIGPIPE in Python 2 (the question has [tag:python2.7] tag). – jfs Dec 23 '14 at 20:37
here's an example of why the redirection is important: it may allow a parent process to exit before its child finished writing its output: [Python subprocess .check_call vs .check_output](https://stackoverflow.com/q/36169571/4279) – jfs Nov 14 '20 at 06:22

score 10 · Answer 2 · answered Dec 23 '14 at 17:40

10

You can use os.fork().

import os
pid=os.fork()
if pid==0: # new process
    os.system("nohup python ./myfile.py &")
    exit()
# parent process continues

answered Dec 23 '14 at 17:40

Hannes Karppila

969
2
13
31

11

Note that `os.fork()` [does not exist for Windows](https://stackoverflow.com/questions/17300551/how-to-run-python-script-with-os-fork-on-windows) – Pro Q Oct 13 '18 at 08:34
Using `nohup` here is pointless. It does exactly two things: (1) disabling HUP propagation, which is **already** turned off for noninteractive shells and so doesn't need to be further disabled; and (2) redirecting stdin, stdout and stderr if they're attached to the terminal (which you can do yourself trivially; `nohup.out 2>&1` and you just did the exact same redirections nohup would do). Much more efficient to just leave all the shells out of it entirely, as the answer by @jfs does. – Charles Duffy Jan 19 '21 at 01:47

score 9 · Answer 3 · edited Sep 19 '19 at 09:06

I could not see any process running.

You don't see any process running because the child python process exits immediately. The Popen arguments are incorrect as user4815162342 says in the comment.

To launch a completely independent process, you could use python-daemon package or use systemd/supervisord/etc:

#!/usr/bin/python25
import daemon
from long_process import main

with daemon.DaemonContext():
    main()

Though it might be enough in your case, to start the child with correct Popen arguments:

with open(os.devnull, 'r+b', 0) as DEVNULL:
    p = Popen(['/usr/bin/python25', '/path/to/long_process.py'],
              stdin=DEVNULL, stdout=DEVNULL, stderr=STDOUT, close_fds=True)
time.sleep(1) # give it a second to launch
if p.poll(): # the process already finished and it has nonzero exit code
    sys.exit(p.returncode)

If the child process doesn't require python2.5 then you could use sys.executable instead (to use the same Python version as the parent).

Note: the code closes DEVNULL in the parent without waiting for the child process to finish (it has no effect on the child).

Launch a completely independent process

3 Answers3

Linked

Related