2
for host in hosts:
    cmd = ['rsh', host, 'myscript']
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = p.communicate()

currently the above hangs on stdout, stderr = p.communicate() because myscript runs forever.

What can i do to make p.communicate() nonblock? i.e. just run the command and move on

ealeon
  • 12,074
  • 24
  • 92
  • 173
  • Why `communicate()` at all, if you're continually overwriting the `stdout` and `stderr` variables with the next element in your loop? – Charles Duffy Jan 08 '16 at 19:07

1 Answers1

3

communicate() has to block, since it asks for stdout and stderr to be returned as strings (and is also explicitly defined to wait for termination). As long as the remote process is running without those pipelines being closed, there still could be more content to add to those strings, so they can't yet be returned.

Eliminate communicate() (and the pipes, since they can fill up if never read from and cause blocking in that way):

processes = [ subprocess.Popen(['rsh', host, 'myscript']) for host in hosts ]
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Okay thanks! my understanding was that 'p' doesnt actually execute the cmd until communicate is called – ealeon Jan 08 '16 at 19:10
  • ...for the use pattern here, you could even eliminate `p` outright, and make the whole line `subprocess.Popen(cmd)`. – Charles Duffy Jan 08 '16 at 20:31
  • Cleaned it up to suggest a pattern that keeps all processes referenced and thus not vulnerable to garbage collection. – Charles Duffy Jan 08 '16 at 20:33
  • 2
    @ealeon: the understanding is incorrect. `Popen()` starts the child process (fork() + exec() on POSIX) -- `.communicate()` is not necessary for that. If you want [to get output from several processes concurrently, you could use threads, async io](http://stackoverflow.com/a/23616229/4279). See whether [`fabric` can be useful in your case, to run remote commands](http://www.fabfile.org) – jfs Jan 09 '16 at 09:57