0

I have several functions that need to run in parallel and online. Meaning, starting one function, and before it finishes starting another one. I found this SO question that works if I want to start the functions simultaneously (and not online):

from multiprocessing import Process

def func1():
  print 'func1: starting'
  for i in range(10000000): pass
  print 'func1: finishing'

def func2():
  print 'func2: starting'
  for i in range(10000000): pass
  print 'func2: finishing'

if __name__ == '__main__':
  p1 = Process(target=func1)
  p1.start()
  p2 = Process(target=func2)
  p2.start()
  p1.join()
  p2.join()

I tried to modify it for the functions to run online:

from multiprocessing import Process

def func1():
    print ('func1: starting')
    for i in range(300000000): pass
    print ('func1: finishing')

def func2():
    print ('func2: starting')
    for i in range(300000000): pass
    print ('func2: finishing')

c = 0

if c < 1:
    p1 = Process(target=func1)
    p1.start()
    p1.join()

c = 20

if c > 19:
    p2 = Process(target=func2)
    p2.start()
    p2.join()

However, it seems like func1 must finish first before func2 starts

Update: Thanks to @Alexey Larionov I think I solved it. I commented out the .join() parts and now the functions run online:

from multiprocessing import Process

def func1():
    print ('func1: starting')
    for i in range(10000000): pass
    print ('func1: finishing')

def func2():
    print ('func2: starting')
    for i in range(100): pass
    print ('func2: finishing')

    
def func3():
    print ('\nfunc3: starting')
    for i in range(100000000): pass
    print ('func3: finishing')

c = 0

if c < 1:
    p1 = Process(target=func1)
    p1.start()
    #p1.join()

c = 20

if c > 19:
    p2 = Process(target=func2)
    p2.start()
    #p2.join()
    
c = 100
    
if c > 99:
    p3 = Process(target=func3)
    p3.start()

>>> func1: starting
func2: starting
func2: finishing
func3: starting

func1: finishing
func3: finishing
Penguin
  • 1,923
  • 3
  • 21
  • 51
  • As soon as you call `p1.join()` that where you say "don't continue until `func1` is finished". You want to `.start()` both processes, store processes in a list, then at the end call `join` for each element of the list. This way you say "now that I've launched all the functions, wait for all of them to finish" – Alexey S. Larionov Jan 08 '21 at 14:50
  • Ahh I see. However, I was looking for a way to start one process, let it run, and at some point run another one, in addition to the first. Maybe I'm misunderstanding your explanation, but it seems like what you suggested refers to the first method I found, that is not online – Penguin Jan 08 '21 at 14:55
  • What do you mean by "online"? If you don't call `p1.join()` then your code will continue to execute further, while `func1` is also being executed in parallel. When you're ready to handle results of `func1`, that's where you call `join` – Alexey S. Larionov Jan 08 '21 at 14:58
  • Oh, I think I'm following now, my apologies. Let me know if what I did was wrong, but I just commented out the `p1.join()` and `p2.join()` in my modified code above and it seems like `func1` starts, then `func2` starts, and then they finish. So can I just not call the `.join` method and continuously start more functions while the previous ones either are running or finished already? – Penguin Jan 08 '21 at 15:03
  • Sure you can, but keep in mind that there is no way to tell the order of when your spawned processes will finish (your main Python program that spawned them may finish earlier than the processing is done, so it may cause immediate termination of all spawned processes). So you're free to spawn a thousand processes, but you better to join them eventually. – Alexey S. Larionov Jan 08 '21 at 15:08
  • I'm not too concerned about the order of when they'll finish, but I didn't quite understand the part about the Python program terminating all of the processes. Can you elaborate on that? Also, I just wrote an update – Penguin Jan 08 '21 at 17:04
  • When you spawn 3 processes, you have 4 processes running in fact (3 are spawned, and the 1 that was running before they were spawned, and it's still running. However this process has its end - the end of you program (end of script etc). In many programming languages it's typical that as soon as the main process is ended all the spawned processes should be killed without waiting them to finish. But I can't find more info if this is the case in Python. From your update it appears that they're not terminated (i.e. they keep on running even after the code of script is done) – Alexey S. Larionov Jan 08 '21 at 20:37
  • Oh I see. That's a really good thing to keep in mind. I appreciate all the help! And yea, it seems like it worked and didn't terminate (maybe because it's Python?) – Penguin Jan 08 '21 at 20:53

0 Answers0