Python - Using Popen.communicate() does not capture the stdout and stderr of a system call as expected

Question

I have a Python script which executes a system call (the call is actually a java program). I am using Popen.communicate() to attempt to capture the stdout and stderr. However, it only works for some Java applications (of which, ALL the java applications I'm calling do display something to the screen when executed directly in the shell).

The code I am using is as follows:

    # Perform the java call
    try:

        p = subprocess.Popen(args,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)

    except OSError as e:
        error_msg = '  Error reading file while executing system command: {0}'.format(e.message)

    except ValueError as e:
        error_msg = '  Error in arugments while executing system command: {0}'.format(e.message)

    else:
        # Display the results in real time
        for line in iter(p.stdout.readline, ''):
            sys.stdout.write(line)

        # Grab the stdout and stderr from the process.
        output, err = p.communicate()

    finally:
        # Check to see if an internal error occurred. If not, flip the flag
        if not err and not error_msg:
            java_completed = True
        else:
            java_completed = False

In some cases, the line sys.stdout.write(line) is never called, and in other's it is.

Does anyone have any experience with this?

score 1 · Answer 1 · answered Mar 18 '14 at 15:08

1

The first thing I notice is that you're reading everything from the process stdout, and then trying to do communicate afterwards: you've already read everything from the pipe, so the output result from p.communicate() will always be empty.

But that doesn't explain why sometimes the sys.stdout.write is not being called.

One thing to keep in mind is that there are multiple ways to display things in a console (i.e., in a command window). Printing to the processes stdout stream is only one way, and that's the way that you'll be able to capture with the above code. If I recall from Java days, there is some type of Console class that is sometimes used for displaying to the console, this may not use stdout. Also, advanced console display libraries, like curses for instance, don't generally write to stdout, they interact with the console object directly. I don't know how, or if, you can capture output from these types of processes, but if it is possible it would probably be very system-specific.

answered Mar 18 '14 at 15:08

brianmearns

9,581
10
52
79

Thanks for the pointers. I think the reason I didn't notice that `output` was empty is that I'm really only curious about the `stderr`. But you certainly helped me learn something about how this works. Unfortunately, the question about how to grab output from the `Console` class is still up in the air. – Brett Mar 18 '14 at 15:14
+1. @Brett: another issue is that you set `stderr=PIPE` but you don't read from it concurrently with `stdout`. It may lead to a deadlock if the program fill the OS pipe buffer corresponding to subprocess' stderr – jfs Mar 18 '14 at 15:22
@J.F.Sebastian Interesting. Any pointers on how I should handle this? – Brett Mar 18 '14 at 15:27
@Brett: you could try `pexpect`, `pty` modules e.g., [Capture “Segmentation fault” message for a crashed subprocess: no out and err after a call to communicate()](http://stackoverflow.com/a/22253472/4279) (it fixes the issue if the program writes directly to the terminal). – jfs Mar 18 '14 at 15:29
@Brett: if you meant how to read both `stdout` and `stderr` concurrently then 1. set `stderr=STDOUT` if it is unacceptable then 2. use threads, or select.select, or fctnl (to make pipes non-blocking), or iocp (on Windows) – jfs Mar 18 '14 at 15:34

Python - Using Popen.communicate() does not capture the stdout and stderr of a system call as expected

1 Answers1