Extract stdout from long-running process

Question

Disclaimer: I have seen many similar questions, but either they do not use asyncio, or they don't seem to do exactly what I do.

I am trying to get the output of a long-running command (it's actually a server that logs out to stdout) with the following:

    proc = await asyncio.create_subprocess_shell(
        "my_long_running_command",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        limit=10
    )

    stdout = ""
    try:
        stdout, stderr = await asyncio.wait_for(proc.communicate(), 2.0)
    except asyncio.exceptions.TimeoutError:
        pass
    print(stdout)

But I don't get anything. If I use ls instead of my_long_running_command, it works. The only difference I see is that ls returns, and not my command.

I am using wait_for because the documentation says:

the communicate() and wait() methods don’t have a timeout parameter: use the wait_for() function;

I tried with limit=10, hoping it would help with the buffering, and without it. It seems to have no effect at all.

Though I don't really understand how they differ, I tried both asyncio.create_subprocess_shell and asyncio.create_subprocess_exec without success.

Is there a way to extract stdout from the process before it returns?

Maybe the buffering could be a problem? See https://stackoverflow.com/questions/1606795/catching-stdout-in-realtime-from-subprocess — Andrej Kesely, Feb 21 '23 at 00:08
Hmm not sure what you suggest, those seem to be using Popen and not asyncio... — JonasVautherin, Feb 21 '23 at 00:17
The I/O buffering is general concept. It doesn't matter if you open the process via Popen or via asyncio: https://stackoverflow.com/questions/1450551/buffered-vs-unbuffered-io — Andrej Kesely, Feb 21 '23 at 00:27
Oh, right. So I tried to set the `limit` argument to a small number (10), but it did not seem to help :/ — JonasVautherin, Feb 21 '23 at 00:29

Andrej Kesely · Accepted Answer · 2023-02-21T23:40:53.453

2

Try to use await proc.stdout.readline() for reading from standard output in realtime. For example:

import asyncio

async def main():
    proc = await asyncio.create_subprocess_exec(
        "top",                        # <--- some long running task
        stdout=asyncio.subprocess.PIPE,
    )

    line = await proc.stdout.readline()
    while line:
        print(line.decode())
        line = await proc.stdout.readline()


asyncio.run(main())

EDIT: Using await proc.stdout.read():

import asyncio

async def main():
    proc = await asyncio.create_subprocess_exec(
        "watch", "-n", "2", "ls", "-alF", "/",
        stdout=asyncio.subprocess.PIPE,
    )

    char = await proc.stdout.read(1)
    while char:
        print(char.decode(), end='')
        char = await proc.stdout.read(1)


asyncio.run(main())

EDIT 2: When two above solutions don't work, try to turn-off the buffering in terminal:

stdbuf -o0 <your command here>

Also, running this Python script without buffering. (for example, using the -u python command line flag)

edited Feb 21 '23 at 23:40

answered Feb 21 '23 at 00:42

Andrej Kesely

168,389
15
48
91

Interestingly, it works with top, but not with my binary... though if I comment out `stdout=asyncio.subprocess.PIPE,`, then my program prints output on stdout... My binary being a static executable, `mavsdk_server_musl_x86_64` from https://github.com/mavlink/MAVSDK/releases/tag/v1.4.12. Could it be doing something weird with stdout? Not sure what to look for :/ – JonasVautherin Feb 21 '23 at 00:58
@JonasVautherin Maybe the program doesn't send newlines (`\n`). I've updated my answer, where I read the stdout one byt at the time. – Andrej Kesely Feb 21 '23 at 01:09
Works for watch, not for `mavsdk_server` :-/. It's weird because really it starts by sending a few lines to stdout (with version number, etc), so there must be something to read. – JonasVautherin Feb 21 '23 at 11:40
1

@JonasVautherin I'm now just guessing, it might be something with buffering. Try to add `stdbuf -o0` before the command. Source: https://www.baeldung.com/linux/stdbuf-pipe-turn-off-buffer . And when you run this python script, try to turn Python buffering off too (`-u` switch) https://stackoverflow.com/questions/107705/disable-output-buffering – Andrej Kesely Feb 21 '23 at 12:07
More than happy to get your guesses, because I'm out of ideas :D. Let me try this! – JonasVautherin Feb 21 '23 at 12:57
1

It works with `stdbuf -o0`!!!!! I'm using your `readline()` example above now, with `stdbuf -o0`. – JonasVautherin Feb 21 '23 at 13:33
1

(please just add `stdbuf` to your answer and I'll mark it as solved ;) ) – JonasVautherin Feb 21 '23 at 23:30

Extract stdout from long-running process

1 Answers1