issue in using subprocess to execute two process simultaneously

Question

I am trying to use subprocess to execute a python script from within a python script but I am having certain issues. Here is what I want to do:

I want to start a main process first (execute python script 1) and after some time into the execution of this process I want to start a subprocess (execute python script 2). Now while this sub process is in execution I wan that the execution of main process also continues and when main process finishes it should wait for the sub process to finish.

Below is the code which I wrote. Here Script1.py is main process script which I import into my code. Script2.py is the sub process script which is called using subprocess.Popen().

Script1.py

import time

def func():
    print "Start time : %s" % time.ctime()
    time.sleep( 2 )
    print "End time: %s" % time.ctime()
    return 'main process'

Script2.py

import time

def sub():
    count=0
    while count < 5:
        print "Start time : %s" % time.ctime()
        time.sleep(3)
        print "End time: %s" % time.ctime()
        x+=1
    return 'sub process'

if __name__ == '__main__':
   print 'calling function inside sub process'
   subval = sub()

Main_File.py is the script which initiated the first process by importing Script1.py and then starting the sub process also later on

Main_file.py

import subprocess
import sys
import Script1

def func1():

    count=0

    while x < 5:
        code = Script1.func()

        if x == 2:
            print 'calling subprocess'
            sub_result = subprocess.Popen([sys.executable,"./Script2.py"]) # Start the execution of sub process. Main process should keep on executing simultaneously
        x+=1
    print 'Main process done'
    sub_result.wait() # even though main process is done it should wait for sub process to get over
    code = sub_result # Get the value of return statement from Sub process
    return code


if __name__ == '__main__':
    print 'starting main process'
    return_stat = func1()
    print return_stat

When I run Main_file.py then the output it executes is not correct. It seems it does not execute the subprocess as I do not see any of print statement written in Script2.py and it stops after main process is done. Also I am not sure about the way of getting the value of the return statement from the sub process. Can anyone help me in trying achieve the correct output.

NOTE: I am new to python and subprocess and so I tried on my behalf first. Please forgive if there is lack of any understanding of concepts

it looks like a buffering issue. [Run with `-u` option to force unbuffered output (both the main process and the subprocess) or just define `PYTHONUNBUFFERED` envvar](http://stackoverflow.com/q/107705/4279) — jfs, Mar 02 '15 at 19:11
@J.F.Sebastian I think it is not entering the subprocess script (Script2.py) as I do not see any of the print statements from that script. Also the execution stops after the execution of Script1.py is over. — Jason Donnald, Mar 02 '15 at 19:13
`python -u Main_file.py`. Don't forget to call `sub()` in `Script2.py` as @Daniel suggested (after `if __name__=="__main__":` guard). Also, importing the script is a more flexible solution (you could use either threading or multiprocessing modules here if necessary to run the code concurrently). — jfs, Mar 02 '15 at 19:16
I've updated indentation of your Python code. Check that it corresponds to your actual code that you use. [Indentation is significant in Python](http://www.secnetix.de/olli/Python/block_indentation.hawk) — jfs, Mar 02 '15 at 19:36

score 1 · Answer 1 · answered Mar 02 '15 at 19:14

1

Subprocess calls an external program. Your Script2 doesn't do anything, because the function sub is not called. Perhaps you want to use threads:

import threading
import Script1
import Script2

def func():
    thread1 = threading.Thread(target=Script1.func)
    thread1.start()
    thread2 = threading.Thread(target=Script2.sub)
    thread2.start()
    thread2.wait()

answered Mar 02 '15 at 19:14

Daniel

42,087
4
55
81

Is it possible to do it through `subprocess` as it is a requirement to make it generic in the application? – Jason Donnald Mar 02 '15 at 19:18
you should probably add `if __name__=="__main__": func()` at the end. – jfs Mar 02 '15 at 19:19
@J.F.Sebastian at end of which script should I mention this? I have added `if __name__=="__main__": ` at then end of `Script2.py` and called `sub()` from there and same for `Script1.py` also. Also how can I get the return value from Script2.py function (the subprocess)? – Jason Donnald Mar 02 '15 at 19:27
If you run the code as is from @Daniel's answer then it just imports the modules and defines the function. It does not actually run it. You need the `__main__` guard only if you want to both run the script as a program and import it in another script. The `__main__` part should be in a file that you want to run as a program. – jfs Mar 02 '15 at 19:31
@J.F.Sebastian THanks for your explanation. How do I get the value of the return statement from the sub process (Scrip2.py)? – Jason Donnald Mar 02 '15 at 19:47
@JasonDonnald: it is best to keep one issue per question. Do you see the expected output now? Daniel already answered how you should arrange the code to get the returned value: import module, call the function (almost -- write a wrapper that puts the returned value into the queue that the main thread can access) – jfs Mar 02 '15 at 20:04
@J.F.Sebastian Yes I was able to get the execution right but not complete as I am not able to read the return value from sub process. I did saw @Daniel's answer but it uses `threads` and I want to use `subprocess`. Is this not possible through `subprocess`? Sorry for troubles! – Jason Donnald Mar 02 '15 at 20:07
@JasonDonnald: the right way is to import the module and call the function directly. If you need a separate process then replace `threading.Thread` with `multiprocessing.Process` in the above code -- don't forget to import `multiprocessing` module first. If you need `subprocess` module specifically then you've to use IPC methods explicitly such as pipes, fifo, sockets, even files. IPC part is better to ask as a separate question – jfs Mar 02 '15 at 20:16

issue in using subprocess to execute two process simultaneously

1 Answers1