Is it possible that processes created by subprocess module in Python still use up memory (are zombies), although subprocess.Popen.wait() was called?

Question

According to the answers to another question in stackoverflow (how to kill (or avoid) zombie processes with subprocess module) one can avoid zombie processes by using the command subprocess.Popen.wait().

However, when I run the following function perform_sth inside my script a several thousand times, the memory usage of each individual process tends to increase:

For example, the first process only needs 7 MB, but nr. 1000 already 500 MB, until in the end more than 8 GB are used and I have to kill the whole Python script. The process should always use more or less the same amount of memory.

Probably I have a flaw in my function and need to additionally kill the processes?

My code is:

def perform_sth(arg1, arg2):
    import subprocess


    sth_cline = ["sth", "-asequence=%s"%arg1, "-bsequence=%s"]
    process = subprocess.Popen(
                                sth_cline, 
                                stdout = subprocess.PIPE, 
                                stderr = subprocess.PIPE
                                )
    process.wait()
    return

Zombies don't use much memory at all; they are just entries in the kernel process table there purely for the parent to read the information on the process. The process itself is gone. — Martijn Pieters, Nov 15 '13 at 14:58
Oops, I didn't know that. Probably the cause of my memory problem lies somewhere else, most likely in the program I am starting. I will have a closer look at that, thank you! — TabeaKischka, Nov 15 '13 at 15:41
if you want to discard output; you could use `stdout=subprocess.DEVNULL` (`DEVNULL=os.open(os.devnull, os.O_WRONLY)`). Don't use `subprocess.PIPE` unless you want to communicate with the subprocess. — jfs, Nov 16 '13 at 19:14

score 1 · Accepted Answer · edited May 23 '17 at 11:50

Do not use stdout=PIPE if you don't read from the pipe.

Your child process is not zombie (a zombie is a dead process; it needs only a tiny amount of memory, to store its exit status in the process table). Your child process is alive (that is why it is capable of consuming gigabytes of memory).

The OS pipe buffer is probably full and the child process is blocked while trying to write to the pipe. Your parent should drain the buffer by reading from the pipe to allow the child to continue but the parent waits for process.wait() to return forever (a deadlock).

If you don't need the output, use stdout=subprocess.DEVNULL instead. Or see How to hide output of subprocess in Python 2.7

#!/usr/bin/env python
from subprocess import check_call, DEVNULL, STDOUT

check_call(["sth", "arg 1", "arg2"], stdin=DEVNULL, stdout=DEVNULL, stderr=STDOUT)

Is it possible that processes created by subprocess module in Python still use up memory (are zombies), although subprocess.Popen.wait() was called?

1 Answers1