1

Suppose I have a python function foo. I want to run it in the background (using threading or multiprocessing), and have all its stdout and stderr available in real time as it is produced by foo. If foo was a shell command, I could achieve this by using subprocess.Popen and a generator, as described here, but I need to do this with a python function. The multiprocessing module is good for spawning new processes, but it seems to be hard to get to child's stdout in real time.

P.S: I need this to redirect function's stdout to a webpage as it is generated, using Server Sent Events protocol

Community
  • 1
  • 1
dmytro
  • 1,293
  • 9
  • 21

2 Answers2

1

For what it's worth, I think this is a bad idea and you should ideally rewrite the foo function to return its output in some other way. That said...

If foo is just a standard pure-Python function using only standard Python functions to produce its output (print and file.write), then you could you just redirect its standard output to a StringIO or something similar.

Python 2.x example:

import sys, StringIO

def foo():
    print "Haha! You can't capture this output!"
    print>>sys.stderr, "... Or this!"

output = StringIO.StringIO()

saved, sys.stdout = sys.stdout, output
sys.stderr = output
foo()
sys.stdout = sys.stderr = saved

print output.getvalue()
Dan Lenski
  • 76,929
  • 13
  • 76
  • 124
  • "I think this is a bad idea". Well, I'm writing a framework, which is supposed to run user code (function `foo` that is) in a separate process or thread, which may print smth to stdout. I don't know of any better solution, but better design suggestions are wellcome – dmytro Jun 19 '14 at 11:24
  • To the actual answer, the `foo` should also be ran in the background. I guess it's not trivial to combine these two techniques, or is it? – dmytro Jun 19 '14 at 12:50
  • 1
    @dymtro: Okay, I think I have a better idea of what you're trying to do. Forking into a child process and then capturing its `stdout` is a fairly common pattern in C programs on POSIX-like systems. It's an OS-dependent, low-level operation. You should be able to implement the procedure outlined in this guide, ["Capture the output of a child process in C"](http://www.microhowto.info/howto/capture_the_output_of_a_child_process_in_c.html), in Python rather than in C, by using `os.pipe`. – Dan Lenski Jun 19 '14 at 16:35
0

You cannot redirect the output of only a function -- you will instead end up redirecting the output of the entire process -- and if that process is threaded then that means every thread's output is now being redirected.

You should check out generators if you are looking for real-time feedback from a single function.

Ethan Furman
  • 63,992
  • 20
  • 159
  • 237
  • As I mentioned in the question, my idea was exactly to spawn a new process for this function. But then I don't know how to do that without having that function in a separate script – dmytro Jun 19 '14 at 11:22
  • @dmytro: In your question you mentioned a new thread as a possibility -- thread's (in Python) are not new processes. You could use multiprocessing and os.pipe, though. – Ethan Furman Jun 19 '14 at 16:40