Intercepting assignments to sys.stdout and sys.stderr

Question

The sys module has a couple of global properties that I'm interested in: sys.stdout and sys.stderr.

I'm building a module of my own, that (among other things), replaces sys.stdout and sys.stderr with its own wrappers that intercept attempted output, modify it, and then forward it to the originals. My method for doing so is something like this:

_orig_stdout = sys.stdout
_orig_stderr = sys.stderr
sys.stdout = MyFakeStdoutClass()
sys.stderr = MyFaleStderrClass()

This works as expected - at any time after my module is imported, trying to do anything with sys.stdout or sys.stderr goes through my class instead.

Now, my module has a vested interest in making sure that this arrangement stays how it is now - it wants to keep control of sys.stdout and sys.stderr permanently. Other modules can reassign sys.stdout the same way that my module did, and my module doesn't want them to be able to do that. Instead, my module wants to intercept their attempt to do so.

For a regular class, doing this would be easy - I'd just have to overwrite the class's __setattr__() method:

_orig_setattr = OtherClass.__setattr__
def my_setattr(obj, name, value):
    if name != "stdout":
        _orig_setattr(obj, name, value)
OtherClass.__setattr__ = my_setattr

However, I've tried doing this for the sys module itself, and it doesn't work (even after I do sys.__setattr__ = my_setattr, I find that my_setattr never gets called).

Additionally, while other answers point out the possible solution of making my own wrapper class for the sys module and assigning it to sys.modules['sys'], this won't work - if sys was imported in another module before my module is imported by that module (which is likely), then my change won't stick.

Furthermore, setting sys.stdout = property(stdout_getter, stdout_setter) with some helper methods to return/modify my _orig_stdout variable didn't work either. Even later in the same file, I could just do sys.stdout = sys.__stdout__, and it was back to normal. I don't want this to be possible.

Is there a good way around this limitation?

@kmaork I would want this code to be platform-independent, usable on anything that runs python 3 (reasonably recent versions thereof). — Green Cloak Guy, Jul 13 '19 at 22:53
Even if you intercepted attribute assignment (and even if you intercepted attribute lookup), you wouldn't be able to intercept direct access to the `sys` module's dict. (Intercepting attribute lookup could block `sys.__dict__`, but not `types.ModuleType.__dict__['__dict__'].__get__(sys, type(sys))`, and not all the C-level code that goes straight for the dict without bothering with attribute lookup.) — user2357112, Jul 14 '19 at 00:05
@user2357112 That's a fair point, but I'm more concerned about other, unrelated modules changing `sys.stdout` themselves for some reason. One of the coolest things about python is that *everything* is changeable, but the things that aren't supposed to be are hard enough to change that it's impossible to break them by accident. That's what I'm striving for here. — Green Cloak Guy, Jul 14 '19 at 00:19
@GreenCloakGuy What exactly are you trying to insert with this `MyFakeStdoutClass()`. It seems to me that in order to get the level of control you want you might need to fully replace the `sys` module. — 9716278, Jul 14 '19 at 01:02
@9716278 It would essentially be modifying any `.write()` statements as they pass through either `sys.stdout` or `sys.stderr`. I'm fine with fully replacing `sys` if I have to, but I can't figure out a way to actually do so and have it stick for the entire program this module is imported by. — Green Cloak Guy, Jul 14 '19 at 01:16
@GreenCloakGuy Could you please add to your question how you want to change the out put of `stdout` and `stderr`? — 9716278, Jul 14 '19 at 04:18
@9716278 I don't think that's especially relevant, actually - in *particular*, I want to intercept every line written to `stdout`/`stderr` and then not write it if a certain flag is set within my module - or replacing text if a different flag is set. I wouldn't think it changes what you'd need to do. — Green Cloak Guy, Jul 14 '19 at 15:11

score 1 · Answer 1 · answered Jul 16 '19 at 23:41

The only real way to completely override stdout on python is to actually override the stdout file descriptor (1). This can be done using the dup2 syscall.

Below is a cross-platform example showing how to override stdout, allowing to use custom logic on all data written to it. In this example, the logic just duplicates all characters written to stdout.

import os
import sys
import threading

def handle_stdout(fake_stdout, real_stdout):
    while not fake_stdout.closed and not real_stdout.closed:
        char = fake_stdout.read(1)
        real_stdout.write(char * 2)

def override_stdout():
    stdout_fd = 1
    pipe_reader_fd, pipe_writer_fd = os.pipe()
    pipe_reader = os.fdopen(pipe_reader_fd, 'r')
    original_stdout_writer = os.fdopen(os.dup(stdout_fd), 'w')
    os.dup2(pipe_writer_fd, stdout_fd)
    return pipe_reader, original_stdout_writer

# Override stdout
pipe_reader, original_stdout_writer = override_stdout()
thread = threading.Thread(target=handle_stdout, args=(pipe_reader, original_stdout_writer))
thread.start()

# Write stuff to stdout
print('foobar')

# Cleanup to allow background thread to shut down
pipe_reader.close()
original_stdout_writer.close()

Running this example will output:

ffoooobbaarr

For some versions of python, you'll have to set the PYTHONLEGACYWINDOWSSTDIO environment variable to a non-empty string to make this example to work on windows.

This looks like it's roughly what I'm looking for, after some experimentation. As a follow-up question, do you know if it's possible to make it so that doing `os.fdopen(pipe_reader_fd)` returns a custom subclass of `TextIOWrapper` instead of the base version defined in `_io`? Thus allowing me to handle that captured output automatically, in a custom way, without having to run a whole new thread to do so? — Green Cloak Guy, Jul 19 '19 at 01:49
To truly override the stdout you need to replace it with a writable fd. `os.pipe` is an easy way to use this fd from within your program. Now the OS buffers stdout for you, and you can use it as you will. If you want to process it in parallel to your code, you'd have to open a thread, use an ioloop or ask the OS to invoke a callback when data is received. As far as I know, there is no good cross platform way to do that. On Linux you could use the aio feature, check out the `pyaio` package on pypi. — kmaork, Jul 19 '19 at 11:21

Intercepting assignments to sys.stdout and sys.stderr

1 Answers1

Linked