2

In Python 3, how can I:

  1. Execute a process and have its stdout printed to stdout,
  2. Monitor the stdout to wait for some text, and
  3. Continue the execution of my Python program without killing the process once the text is found and without stopping it printing to stdout.

live output from subprocess command is not a duplicate because all of the answers there do not allow your Python code to continue and have the process's stdout still be printed.

I imagine the solution will involve some kind of thread, coroutine or task.

Timmmm
  • 88,195
  • 71
  • 364
  • 509
  • Bit of an indirect approach so just gonna comment, but what if you fork at the os layer? Execute the process with an & flag? Don't know if it will work. Just an idea. https://bashitout.com/2013/05/18/Ampersands-on-the-command-line.html – Neil May 21 '20 at 15:40
  • Exact duplicate [Execute subprocess that prints to stdout and wait for text, while allowing continued execution after text is seen](https://stackoverflow.com/questions/61937067/execute-subprocess-that-prints-to-stdout-and-wait-for-text-while-allowing-conti) – jordanm May 21 '20 at 15:40
  • @jordanm: StackOverflow specifically says you can open a new question if a question is closed. Please reconsider whether your actions are useful. – Timmmm May 21 '20 at 15:44
  • @Neil: Interesting idea but I don't think it will work because I still have to process the `stdout` data somehow. Also I'd rather not run my command through the shell (generally a bad idea). – Timmmm May 21 '20 at 15:46
  • @Timmmm I didn't realize policies had changed. In the past, it was intended that you edit your existing question and vote to reopen. The close -> duplicate button used to actually say "exact duplicate" – jordanm May 21 '20 at 15:56
  • Anways, I would start with the orginal thing I linked. Try doing `p.stdout = sys.stdout` and break the loop to continue output to stdout – jordanm May 21 '20 at 15:58
  • Ahh interesting idea, I wouldn't have thought that would work but I will try it. – Timmmm May 21 '20 at 16:00
  • Sadly it didn't work. I think what you really need is something like `p.stdout.pipe(sys.stdout)` but I haven't found anything that does that yet. – Timmmm May 21 '20 at 16:20

1 Answers1

0

A relatively simple approach is to start a new thread to copy the process's stdout to Python's stdout after you are finished processing it. Here is an example for running webpack-dev-server and scraping the URL from it (not actually implemented):

#!/usr/bin/env python3

import subprocess
import sys
import os
from threading import Thread

from typing import IO

def pipe(a: IO[bytes], b: IO[bytes]):
  for x in a:
    b.write(x)
    b.flush()


process = subprocess.Popen(
  [
    "node_modules/.bin/webpack-dev-server",
    # Note that I had to add --colour because for some reason
    # webpack-dev-server detects stdout as supporting colour,
    # but not stderr.
    "--color",
    # This prints a progress bar using ANSI escape characters
    # which works too!
    "--progress",
  ],
  cwd=".",
  env=dict(os.environ, NODE_ENV="development"),
  stdout=subprocess.PIPE,
)

for line in process.stdout:
  sys.stdout.buffer.write(line)
  # Flush is necessary otherwise stdout will buffer more
  # than one line.
  sys.stdout.buffer.flush()

  # Process the text however you want.
  if b"Project is running at" in line:
    print("Server started at address ??")
    break

# Start a thread to do the pipe copying.
thread = Thread(target=pipe, args=(process.stdout, sys.stdout.buffer))
thread.start()

print("Now we can do other stuff and the process will continue to print to stdout")

Note that I haven't given any thought to proper cleanup - exiting the thread, closing files, etc.

Timmmm
  • 88,195
  • 71
  • 364
  • 509