python subprocess won't interleave stderr and stdout as what terminal does

Question

A test program

#!/usr/bin/env python3

import sys

count = 0
sys.stderr.write('stderr, order %d\n' % count)
count += 1
sys.stdout.write('stdout, order %d\n' % count)
count += 1
sys.stderr.write('stderr, order %d\n' % count)
count += 1
sys.stdout.write('stdout, order %d\n' % count)

when invoked through the terminal, the expected output is,

stderr, order 0
stdout, order 1
stderr, order 2
stdout, order 3

In the interactive shell, when I redirect stdout to a PIPE, the output order is different from the output above, where Popen would group stderr and write them all and then do the same thing to stdout, instead of interleaving stdout and stderr.

In [29]: a = sp.run(['./test.py'], stderr=sp.STDOUT)
stderr, order 0
stdout, order 1
stderr, order 2
stdout, order 3

In [30]: a
Out[30]: CompletedProcess(args=['./test.py'], returncode=0)

In [33]: b = sp.Popen(['./test.py'], stderr=sp.STDOUT, stdout=sp.PIPE, encoding='utf-8')

In [34]: print(b.communicate()[0])
stderr, order 0
stderr, order 2
stdout, order 1
stdout, order 3

tdelaney · Answer 1 · 2020-02-16T22:25:37.953

In the C libraries (and thus c-based python), streams are handled differently depending on whether they are attached to interactive terminals (or something pretending to be) or not. For a tty, stdout is line buffered, otherwise its block buffered and only flushed to the file descriptor when some block boundary is hit. When you redirected to a PIPE, the stream is no longer a tty and block buffering is in effect.

The solution is to reopen stdout specifying that you want line buffering (1) regardless. At the C level, stderr is always line buffered but when I tested just reopening stdout the program acted as though stderr is block buffered. I was quite surprised. Maybe this is the intermediate io.TextIO layer or some other odd thing, but I found I needed to fix both pipes.

Even though stdout and stderr go to the same pipe, they are separate file descriptors with separate buffers as far as the executed program is concerned. That's why interleaving doesn't happen naturally in the output buffer even in block mode.

#!/usr/bin/env python3

import sys
import os

# reopen stdout line buffered
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1)

# this surprises me, seems like we have to reopen stderr
# line buffered, but i thought it was line buffered anywy.
# perhaps its the intermediate python TextIO layer?
sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 1)

count = 0
sys.stderr.write('stderr, order %d\n' % count)
count += 1
sys.stdout.write('stdout, order %d\n' % count)
count += 1
sys.stderr.write('stderr, order %d\n' % count)
count += 1
sys.stdout.write('stdout, order %d\n' % count)

python subprocess won't interleave stderr and stdout as what terminal does

1 Answers1

Linked