Subproces.popen - slave write failed: Broken pipe; aborting

Question

TLDR: stuck with this https://code.google.com/archive/p/byte-unixbench/issues/1

Trying to run UnixBench using subprocess.popen() while capturing output and printing it out in realtime.

This is the subroutine I've come up with:

def run_and_print(command, cwd=None, catch_stderr = False):
    if catch_stderr:
        err_pipe = subprocess.PIPE
    else:
        err_pipe = subprocess.STDOUT

    p = subprocess.Popen(command, stdout=subprocess.PIPE, bufsize=1, cwd=cwd, stderr=err_pipe)
    r = ''
    while True:
        if catch_stderr:
            out = p.stderr.read(1)
        else:
            out = p.stdout.read(1)
        if out == "" and p.poll() != None:
            break
        sys.stdout.write(out)
        sys.stdout.flush()
        r += out

    return r

It works just fine for all the purposes except for UnixBench. Unixbench just dies after a while:

unixbench = run_and_print(['./Run'])

...

1 x Pipe Throughput 1 2 3 4 5 6 7 8 9 10

1 x Pipe-based Context Switching 1 2 3 4

Run: "Pipe-based Context Switching": slave write failed: Broken pipe; aborting

Google didn't help much. The only meaningful result I've got is https://code.google.com/archive/p/byte-unixbench/issues/1 and suggest solution to create a java app won't work for me as I need to run the script with as few dependencies as possible.

I'll be thankful for any solution or a workaround. The system I'm testing this on is Ubuntu 14.04.4 x64

Probably won't help, but did you try `p.communicate()`? One possible hack: don't try to capture the output in Python; instead, run the command by redirecting stderr and stdout to different text files. Then read those files via Python. — FMc, Jun 06 '16 at 03:13
@FMc Great idea about writing to files, not sure how to print stdout in realtime if it's redirected to a file. How would you suggest using p.communicate()? It was my understanding that with `communicate` I won't get any output until the process is done. Thanks for the reply. — Anton, Jun 06 '16 at 03:24
@user1556912, your understanding is correct, you cannot use communicate if you want the output immediately. — Padraic Cunningham, Jun 06 '16 at 09:52
related: [Run command and get its stdout, stderr separately in near real time like in a terminal](http://stackoverflow.com/a/31953436/4279) and [Displaying subprocess output to stdout and redirecting it](http://stackoverflow.com/q/25750468/4279) — jfs, Jun 07 '16 at 14:07

score 2 · Accepted Answer · edited May 23 '17 at 11:52

The bug is related to 'yes' reporting error with subprocess communicate() which provides the fix: reenable SIGPIPE signal in the child process using preexec_fn (or use Python 3).

Unrelated: your code can deadlock if catch_stderr is true and p.stderr and p.stdout are not perfectly in sync.

Otherwise catch_stderr has no effect (ignoring buffering): your code captures stderr regardless. You could simplify it:

#!/usr/bin/env python
from shutil import copyfileobj
from subprocess import Popen, PIPE, STDOUT

def run_and_print(command, cwd=None):
    p = Popen(command, stdout=PIPE, stderr=STDOUT, bufsize=-1, cwd=cwd,
              preexec_fn=restore_signals)
    with p.stdout:
        tee = Tee()
        copyfileobj(p.stdout, tee)        
    return p.wait(), tee.getvalue()

where Tee() is a file-like object that writes to two places: to stdout and to StringIO():

import sys
from io import BytesIO

class Tee:
   def __init__(self):
       self.file = BytesIO()
   def write(self, data):
       getattr(sys.stdout, 'buffer', sys.stdout).write(data)
       self.file.write(data)
   def getvalue(self):
       return self.file.getvalue()

where restore_signals() is defined here.

If you want to see the output on the screen as soon as command prints them; you could inline Tee, copyfileobj() and use os.read(), to avoid reading the complete length before writing it to stdout:

chunks = []
with p.stdout:
    for chunk in iter(lambda: os.read(p.stdout.fileno(), 1 << 13), b''):
        getattr(sys.stdout, 'buffer', sys.stdout).write(chunk)
        sys.stdout.flush()
        chunks.append(chunk)
return p.wait(), b''.join(chunks)

To disable the internal block-buffering in the child process, you might try to run it using stdbuf or pass pseudo-tty instead of the pipe.

Thanks for all of your suggestions! Will report back as soon as I test them out. As for catch_stderr having no effect, sometimes I run commands that print to `stderr` instead of `stdout` (`dd` as an example). I will see if your code solves that as well. Thank you very much. — Anton, Jun 07 '16 at 19:41
I can confirm that `preexec_fn` fixed the issue with the UnixBench. It finishes all the benchmarks successfully. However, I couldn't make your version with `Tee` to output in realtime (the way the original function worked) but I could modify my code to work. I still need to figure out a way around the potential deadlock issue you've pointed out. Thanks again! You're the best! — Anton, Jun 08 '16 at 13:32
Oh and I've noticed your update too late. Thanks for the implementation of real-time output. Will try that tonight! SO needs to implement "send a beer" button :) — Anton, Jun 08 '16 at 13:34
@Anton: The code in the answer captures both stdout/stderr as your code does but without the risk of the deadlock. You could subscribe to the question feed (below on the right) — jfs, Jun 08 '16 at 16:09

Subproces.popen - slave write failed: Broken pipe; aborting

1 Answers1