Python 3: Unbuffered vs Buffered Streams

Question

I have been using the following snippet to silence (redirect output from) C code called in my Python script:

from ctypes import CDLL, c_void_p
import os
import sys

# Code
class silence(object):
    def __init__(self, stdout=os.devnull):
        self.outfile = stdout

    def __enter__(self):
        # Flush
        sys.__stdout__.flush()

        # Save
        self.saved_stream = sys.stdout
        self.fd = sys.stdout.fileno()
        self.saved_fd = os.dup(self.fd)

        # Open the redirect
        self.new_stream = open(self.outfile, 'wb', 0)

        self.new_fd = self.new_stream.fileno()

        # Replace
        os.dup2(self.new_fd, self.fd)

    def __exit__(self, *args):
        # Flush
        self.saved_stream.flush()

        # Restore
        os.dup2(self.saved_fd, self.fd)
        sys.stdout = self.saved_stream

        # Clean up
        self.new_stream.close()
        os.close(self.saved_fd)


# Test case
libc = CDLL('libc.so.6')


# Silence!
with silence():
    libc.printf(b'Hello from C in silence\n')

The idea is to redirect the fd associated with stdout and replace it with one associated with an open null device. Unfortunately, it does not work as expected under Python 3:

$ python2.7 test.py
$ python3.3 -u test.py
$ python3.3 test.py
Hello from C in silence

Under Python 2.7 and 3.3 with unbuffered output it does work. I am unsure what the underlying cause is, however. Even if stdout is buffered the call to sys.saved_stream.flush() should end up calling fflush(stdout) at the C level (flushing the output to the null device).

What part of the Python 3 I/O model am I misunderstanding?

try [`with stdout_redirected():`](http://stackoverflow.com/a/22434262/4279) — jfs, Jan 27 '15 at 08:31
redirect_stdout does not help if the writes to stdout are coming from C library code (such as printf and friends). — Freddie Witherden, Jan 27 '15 at 11:35
read the complete answer (pay attention to the function names) — jfs, Jan 27 '15 at 11:37
Apologies for the typo, my comment was with respect to the stdout_redirected function provided in the answer. It does not handle the case when the C library stdio functions come into the mix with their own brand of buffering (and so suffers from the same issues as the above snippet). — Freddie Witherden, Jan 27 '15 at 13:17
yes, it is likely because Python 3 I/O (unlike Python 2) does not use stdio for standard streams therefore calling `stdout.flush()` might not be enough. Does it help if you add the corresponding `libc.fflush(None)` calls as @Kolmar suggested? If you're only interested in stdio streams then you could use `freopen()` function to redirect standard stdio streams. — jfs, Jan 27 '15 at 13:40
I think `fflush(None)` is a reasonable solution if I can work out/find a portable way of getting a handle to libc from ctypes. — Freddie Witherden, Jan 27 '15 at 13:46
see [how `standard_c_lib` is calculated](http://stackoverflow.com/a/661300/4279) — jfs, Jan 27 '15 at 14:00

score 1 · Accepted Answer · answered Jan 27 '15 at 00:53

1

I'm not 100% sure I understand the Py3 I/O model either, but adding

    sys.stdout = os.fdopen(self.fd, 'wb', 0)

right after your assignment to self.fd fixes it for me in Python 3.4 (I was able to reproduce the problem in 3.4 before I added this statement).

answered Jan 27 '15 at 00:53

Alex Martelli

854,459
170
1,222
1,395

score 0 · Answer 2 · answered Jan 27 '15 at 02:24

I'm not entirely sure what's going on either, but on my system there are two ways to fix this:

Replace the call to self.saved_stream.flush() in __exit__ with libc.fflush(None).
Call libc.printf with any string before calling silence(), for example:
```
libc = CDLL('/bin/cygwin1.dll')
libc.printf(b'')
```

Also, only with the second way has the outputs of Python's print and libc.printf remains synchronized after with silence(): block.

Python 3: Unbuffered vs Buffered Streams

2 Answers2