1

I am trying to write and read from a CLI program using subprocess.Popen.

So every time you issue a command to the CLI it prints something, and that something is what I want to read. I will try to explain what I want do to with a concrete example so everybody can try.

Thus, instead of using my real program I will use ipython for which the problem holds:

p = subprocess.Popen(['ipython'],stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate(input="5+5")

If you use Popen.communicate(input=command) you get at stdout exactly all the text that ipython would print in the terminal if I would have executed it directly from the shell.

What I want to do:

Instead of using communicate. I want to be able to send individual commands and read the immediate output. For example I would like to do:

p.stdin.write("5+5")
out1 = p.stdout.read()

p.stdin.write("5*10")
out2 = p.stdout.read()

So out1 should be 5 and out2 should be 50.

Problem:

You can not read the stdout until you close stdin.

How do you do that ? I can't use read(n) or readline() since in my real application I do not know the length of the output.

edgarstack
  • 1,096
  • 1
  • 10
  • 18
  • Possible duplicate: [Interactive input/output using python.](https://stackoverflow.com/questions/19880190/interactive-input-output-using-python) – John1024 Dec 22 '17 at 23:47
  • Possible duplicate: [How to open a bash session with Python and keep communicating with it?](https://stackoverflow.com/questions/47947625/how-to-open-a-bash-session-with-python-and-keep-communicating-with-it) – John1024 Dec 22 '17 at 23:48
  • @John1024 thanks. It is not the same exactly but maybe it does work. I will give them a try. EDIT: I tried and does not work, somehow does solutions allow you to read without blocking but there is nothing at the output until you do `p.stdin.close()`. – edgarstack Dec 23 '17 at 00:26
  • The other question showed a number of possible solutions. Proper form on StackExchange would be to [edit](https://stackoverflow.com/posts/47948512/edit) your question above precisely what you tried and precisely how it didn't work. That would (a) demonstrate that this is not a duplicate, and (b) give people here something to work with. – John1024 Dec 23 '17 at 00:39
  • I added a new version with an example where the problem happens. – edgarstack Dec 23 '17 at 01:08
  • 2
    You can't `read()` the stdout, as that will read to the end of the file. But you can read bytes or lines, e.g. using `.readline()`. – Daniel Pryden Dec 23 '17 at 01:11
  • 1
    Possible duplicate of [Non-blocking read on a subprocess.PIPE in python](https://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python) – Daniel Pryden Dec 23 '17 at 01:14
  • I tried that but I do not know the length of the output. But now that you say it. Would it work if I combine `readline()` and the non-blocking solution? I read lines until I get the exception. – edgarstack Dec 23 '17 at 01:15
  • Actually that does not work. In the case of the `ipython` you can read line by line. However with my `other-prompt` when you do a `readline()` there is nothing to read apparently. So the program just puts something to `stdout` when you really close `stdin`. But then how is it possible that if you run that program with the normal shell, then its able to print outputs when you "hit" return. – edgarstack Dec 23 '17 at 01:34

2 Answers2

2

With some minor modifications, I was able to make the approach from this question work.

Using Python3:

>>> from subprocess import Popen, PIPE
>>> 
>>> fw = open("tmpout", "wb")
>>> fr = open("tmpout", "r")
>>> p = Popen("ipython", stdin=PIPE, stdout=fw, stderr=fw, bufsize=1, universal_newlines=True)
>>> header = fr.read()
>>> p.stdin.write("100*50\n")
7
>>> fr.read()
'\nIn [1]: Out[1]: \n5000\n'
>>> p.stdin.write("2/3.\n")
5
>>> fr.read()
'\nIn [2]: Out[2]: \n0.6667\n'

As you can see, it works interactively.

A trick is that, under Py3, bufsize=1 makes I/O line-buffered only if universal_newlines=True.

John1024
  • 109,961
  • 14
  • 137
  • 171
  • what about python2? – edgarstack Dec 24 '17 at 00:03
  • @edgarstack The same solution works for me on Python2 with or without the `universal_newlines=True` option. (If you have a choice, use Py3: Py2 is approaching [end-of-life.](https://pythonclock.org/)) – John1024 Dec 24 '17 at 00:10
0

Calling communicate() blocks until the process exits, which is not what you want. You need to read from p.stdout and write to p.stdin.

Alternatively, you might find the pexpect module useful.

Daniel Pryden
  • 59,486
  • 16
  • 97
  • 135
  • Thats what I said somehow. With `communicate` it works because it kills makes the process terminate somehow. But with `p.read()` which is what I want it does not work. – edgarstack Dec 22 '17 at 23:48
  • @edgarstack: You should edit your question to show your code. You might be deadlocking with the other process. – Daniel Pryden Dec 23 '17 at 00:13
  • The other process I can not show it because its a big program. But my code is basically that. 1) `p=popen`, 2) `p.stdin.write(command)`, 3) `p.stdout.read()`. – edgarstack Dec 23 '17 at 00:24