How do I retrieve stderr for a shell command with a large data buffer?

Question

Python subprocess module states regarding the communicate() function:

Note

The data read is buffered in memory, so do not use this method if the data size is large or unlimited.

How can I execute a process that reads a lot of data (e.g. communicate() is contraindicated) and yet still have access to the stderr output?

Are you asking how to read both stdout/stderr without storing the whole output in memory i.e., without `.communicate()`? The simplest portable way would be to use threads e.g., see [`teed_call()`](http://stackoverflow.com/a/4985080/4279) — jfs, Apr 24 '14 at 20:01
I was under the (false) impression that you could only access stderr output if you previously used communicate(). stderr can actually simply be retrieved using dump_proc.stderr.read(). — ChaimKut, Apr 30 '14 at 08:30
the issue with using `proc.stderr.read()` is that if you also use `proc.stdout.read()` then your process may deadlock if it generates enough output to fill the corresponding pipe buffer that is why you need threads, or non-blocking pipes, etc — jfs, Apr 30 '14 at 11:39

score 1 · Answer 1 · answered Apr 24 '14 at 07:46

1

"communicate()" solves this problem by using threads. That means, you need an extra stderr reading thread, while doing the main work (reading stdout) in the main thread. Alternatively you can use select.select, but this doesn't work with windows.

answered Apr 24 '14 at 07:46

Daniel

42,087
4
55
81

score 1 · Accepted Answer · edited May 23 '17 at 11:49

To get possibly unlimited subprocess' stdout/stderr output separately as soon as it becomes available, you could use twisted spawnProcess():

#!/usr/bin/env python
from twisted.internet import protocol
from twisted.internet import reactor

class ProcessProtocol(protocol.ProcessProtocol):
    def outReceived(self, data):
        print 'got stdout:', data
    def errReceived(self, data):
        print 'got stderr:', data
    def processEnded(self, reason):
        reactor.stop()

process = ProcessProtocol()
reactor.spawnProcess(process, 'cmd', ['cmd', 'arg 1', 'arg 2'])
reactor.run()

An alternative is to use threads e.g., teed_call() or use OS specific code e.g., fcntl module to make the pipes non-blocking on POSIX systems or use Overlapped I/O with named pipes on Windows.

score 0 · Answer 3 · answered Apr 24 '14 at 07:56

0

Depending on your type of problem you may be able to rearrange the code to pipe the stderr process into your python code. This page has some pointers.

answered Apr 24 '14 at 07:56

Laur Ivan

4,117
3
38
62

How do I retrieve stderr for a shell command with a large data buffer?

3 Answers3

Linked