Pythonic way to detach a process?

Question

I'm running an etcd process, which stays active until you kill it. (It doesn't provide a daemon mode option.) I want to detach it so I can keep running more python.

What I would do in the shell;

etcd & next_cmd

I'm using python's sh library, at the enthusiastic recommendation of the whole internet. I'd rather not dip into subprocess or Popen, but I haven't found solutions using those either.

What I want;

sh.etcd(detach=True)
sh.next_cmd()

or

sh.etcd("&")
sh.next_cmd()

Unfortunately detach is not a kwarg and sh treats "&" as a flag to etcd.

Am I missing anything here? What's the good way to do this?

Answer 1

To implement sh's &, avoid cargo cult programming and use subprocess module directly:

import subprocess

etcd = subprocess.Popen('etcd') # continue immediately
next_cmd_returncode = subprocess.call('next_cmd') # wait for it
# ... run more python here ...
etcd.terminate() 
etcd.wait()

This ignores exception handling and your talk about "daemon mode" (if you want to implement a daemon in Python; use python-daemon. To run a process as a system service, use whatever your OS provides or a supervisor program such as supervisord).

Answer 2

Author of sh here. I believe you want to use the _bg special keyword parameter http://amoffat.github.io/sh/#background-processes

This will fork your command and return immediately. The process will continue to run even after your script exits.

Answer 3

subprocess is easy enough to do this too:

This approach works (python3). The key is using "start_new_session=True"

UPDATE: despite Popen docs saying this works, it does not. I found by forking the child and then doing os.setsid() it works as I want

client.py:

#!/usr/bin/env python3
import time
import subprocess
subprocess.Popen("python3 child.py", shell=True, start_new_session=True)
i = 0
while True:
    i += 1
    print("demon: %d" % i)
    time.sleep(1)

child.py:

#!/usr/bin/env python3
import time
import subprocess
import os

pid = os.fork()
if (pid == 0):
    os.setsid()

    i = 0
    while True:
        i += 1
        print("child: %d" % i)
        time.sleep(1)
        if i == 10:
            print("child exiting")
            break

output:

./client.py
demon: 1
child: 1
demon: 2
child: 2
^CTraceback (most recent call last):
  File "./client.py", line 9, in <module>
    time.sleep(1)
KeyboardInterrupt

$ child: 3
child: 4
child: 5
child: 6
child: 7
child: 8
child: 9
child: 10
child exiting

Answer 4

Note in the following two examples there is a call to time.sleep(...) to give etcd time to finish starting up before we send it a request. A real solution would probably involving probing the API endpoint to see if it was available and looping if not.

Option 1 (abusing the multiprocessing module):

import sh
import requests
import time

from multiprocessing import Process

etcd = Process(target=sh.etcd)

try:
    # start etcd
    etcd.start()
    time.sleep(3)

    # do other stuff
    r = requests.get('http://localhost:4001/v2/keys/')
    print r.text
finally:
    etcd.terminate()

This uses the multiprocessing module to handle the mechanics of spawning a background tasks. Using this model, you won't see the output from etcd.

Option 2 (tried and true):

import os
import signal
import time
import requests

pid = os.fork()
if pid == 0:
    # start etcd
    os.execvp('etcd', ['etcd'])

try:
    # do other stuff
    time.sleep(3)
    r = requests.get('http://localhost:4001/v2/keys/')
    print r.text
finally:
    os.kill(pid, signal.SIGTERM)

This uses the traditional fork and exec model, which works just as well in Python as it does in C. In this model, the output of etcd will show up on your console, which may or may not be what you want. You can control this by redirecting stdout and stderr in the child process.

Answer 5

Posting this if for no other reason than finding it next time I google the same question:

 if os.fork() == 0:
    os.close(0)
    os.close(1)
    os.close(2)
    subprocess.Popen(('etcd'),close_fds=True)
    sys.exit(0)

Popen close_fds closes the file descriptors other than 0,1,2, so the code closes them explicitly.