python: fork without an external command, and capturing stdout and stderr separately

Question

I'd like to fork a subprocess in python that does not run an external command ... it would just run a defined function. And I want to capture stdout and stderr separately.

I know how to use os.fork() and os.pipe(), but that mechanism only gives me two fd's to work with. I'm looking for three fd's: one for stdin, one for stdout, and one for stderr. This is easy to manage using subprocess.Popen when running an external command, but that function doesn't seem to allow a local function to be forked; only a separate executable.

In ruby, the popen3 command can take "-" as its command argument, and in this case, a fork takes place without any external command being invoked, and the 3 fd's I mentioned are returned. Is there some sort of python analog to this routine in python?

Your question is slightly confusing to me.. What do you mean by: `only gives me two fd's to work with`? Anyways, I have answered your question based upon what I could understand out of it. Let me know if that works! — Vishal, Nov 22 '19 at 03:33
A single call to `os.pipe()` returns two file descriptors ("fd's"). That's what I meant. Anyway, your solution offered below indeed answers my question, and thank you very much. — HippoMan, Nov 22 '19 at 10:49

Vishal · Accepted Answer · 2019-11-22T03:32:32.260

If you want to redirect the stdout and stderr separately from the child process, you can simply create two separate pipes for each, instead of one. I have shared the relevant code.
You can also read this thread to gain more knowledge on this subject: Redirect stdout to a file in Python?
I have mentioned two methods for writing to stdout, and stderr from the child process (Method1, Method2)
If you want to write to stdin of child process as well, you should create another file descriptor. This time the r would go to child process, and the w would go to the parent process.

import os
import sys
import time

# Create two pipes. One for sys.stdout, and one for sys.stderr
r_out, w_out = os.pipe()
r_err, w_err = os.pipe()

pid = os.fork()
if pid == 0:
    # Child process
    os.close(r_out)
    os.close(r_err)

    w1 = os.fdopen(w_out, "w")
    w2 = os.fdopen(w_err, "w")
    sys.stdout = w1
    sys.stderr = w2

    # Note that flush=True is necessary only if you want to ensure the order of messages printed
    # across method1, and method2 is maintained

    # Method 1: Standard Python print messages
    print("Redirected to stdout #2", flush=True)
    print("Redirected to stderr #2", file=sys.stderr, flush=True)

    # Method 2: Using system file descriptors
    stdout_fd = sys.stdout.fileno()
    os.write(stdout_fd, b'Redirected to stdout')

    stderr_fd = sys.stderr.fileno()
    os.write(stderr_fd, b'Redirected to stderr')

    # Restore original stdout, and stderr
    sys.stdout = sys.__stdout__
    sys.stderr = sys.__stderr__

    # Close the file descriptors
    w1.close()
    w2.close()

else:
    # Parent process
    os.close(w_out)
    os.close(w_err)

    r1 = os.fdopen(r_out)
    r2 = os.fdopen(r_err)
    for i in range(5):
        # Note that r1.read(), and r2.read() are non-blocking calls
        # You can run this while loop as long as you want.
        print("Read text (sysout):", r1.read())
        print("Read text (syserr):", r2.read())
        time.sleep(0.5)

python: fork without an external command, and capturing stdout and stderr separately

1 Answers1

Linked