Subprocess PIPE stdout to two different processes

Question

I am trying to redirect a subprocess stdout to two different processes. Doing it to one works fine, but not quite working with two different processes

Code:

def terminate_processes(*args):
    try:
        for i in reversed(args):
            try:
                i.terminate()
            except:
                pass
    except:
        pass

def main(x='192.168.0.2'):


    # system('sudo nmap -O '+x+'|grep "Running: " > os_detect.txt')
    # system('cat os_detect.txt|cut -d " " -f2 > os.txt')
    # system('cat os_detect.txt|cut -d " " -f3 > os_version.txt')
    
    p1 = subprocess.Popen(['sudo', 'nmap', '-O', str(x)], stdout=subprocess.PIPE)
    
    p2 = subprocess.Popen(['grep', 'Running: '],  stdin=p1.stdout, stdout=subprocess.PIPE)
    
    p3 = subprocess.Popen(['cut', '-d', ' ', '-f2'],  stdin=p2.stdout,
                                                    stdout=subprocess.PIPE, 
                                                    stderr=subprocess.STDOUT,
                                                    universal_newlines=True)
    
    p4 = subprocess.Popen(['cut', '-d', ' ', '-f3'],  stdin=p2.stdout,
                                                    stdout=subprocess.PIPE, 
                                                    stderr=subprocess.STDOUT,
                                                    universal_newlines=True)

    while p3.poll() is None:
        for line in p3.stdout.readlines():
            if line.strip():
                print(line.strip())

    while p4.poll() is None:
        for line in p4.stdout.readlines():
            if line.strip():
                print(line.strip())

    terminate_processes(p1,p2,p3,p4)

As i said should work in theory cause works when only using p3 and not p4, but not working in this case maybe because the stdout is locked.

Any guidance would be really appreciated.

And i am reversing the args array inside terminate function cause killing the child process before killing the parent one.

jmunsch · Accepted Answer · 2020-06-22T19:04:41.637

2

The way that .read() generally works, in most cases that I'm aware of, in order to use it a second time, you'd have to use .seek() to rewind the read head back to where it was before.

see:

Why can't I call read() twice on an open file?

What you can do is use communicate and manually pass in the stdout data ( reads once, pass into both):

out, err = p2.communicate() # out is None, since you don't 

p2_output = ''.join(list(out))

p3 = Popen([...], stdin=PIPE, ...)
p4 = Popen([...], stdin=PIPE, ...)

stdout_data3, err = p3.communicate(input=p2_output)
stdout_data4, err = p4.communicate(input=p2_output)

Also note, this might change the way polling needs to be done, in comparison to what you currently have.

answered Jun 22 '20 at 19:01

jmunsch

22,771
11
93
114

is using communicate more efficient than using the stdin=p2.stdout ? – Shantanu Bedajna Jun 22 '20 at 19:04
1

@ShantanuShady i think in this case efficiency is less of a problem, and what is more of a concern is being able to customize the way the data is being passed to get the intended result, something that works, vs something that doesn't. To answer your question directly, I don't know off hand, however, i'd guess that since there is an extra step, or two, there might be a couple of microseconds difference. https://softwareengineering.stackexchange.com/questions/80084/is-premature-optimization-really-the-root-of-all-evil – jmunsch Jun 22 '20 at 19:16
1

thanks , had to decode it was coming as bytes print('out is', out.decode('utf8')) – Shantanu Bedajna Jun 22 '20 at 19:19

Subprocess PIPE stdout to two different processes

1 Answers1