Why close_fds=False sometimes hangs the process in Python 2?

Question

I noticed that in Python3 subprocesses the default for Popen parameter close_fds was changed from False to True and I wondered what was the reason and whether it is a good practice to almost always set close_fds to True (as I'm still using Python 2.7).

I have found a link showing an issue with close_fds=False.

https://bugs.python.org/issue7213

Unfortunately, it isn't clear to me why it so happens.

import subprocess as sub
p1 = sub.Popen(['cat'], stdin=sub.PIPE, stdout=sub.PIPE, close_fds=False)
p2 = sub.Popen(['grep', 'a'], stdin=p1.stdout, stdout=sub.PIPE, close_fds=False)
p1.stdin.write("aaaaaaaaaaaaaaaa\n")
p1.stdin.close()
p2.stdout.read()  # Hangs on Python 2

The program hangs on Python2, does not on Python3 and does not hang at all if close_fds is set to True. So I wonder... What was the actual issue there?

EDIT: It hangs on my Python 2.6 and stopped hanging on 2.7

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

What was the actual issue there?

With Python 2.6 the file handle of the write end of the p1.stdin pipe is inherited by p2, which doesn't care or know about it and so leaves it open. Consequently, although the parent process does p1.stdin.close(), the write pipe remains open and thus the cat doesn't detect EOF on its input and keeps waiting for data from the pipe, blocking the whole process chain.

With Python 2.7 calls of fcntl(…, F_SETFD, FD_CLOEXEC) follow the creation of the stdin and stdout pipes, so that the write pipe end isn't inherited by p2 and effectively closed by p1.stdin.close().

Why close_fds=False sometimes hangs the process in Python 2?

1 Answers1