Timing a multiprocessing script

Question

I've stumbled across a weird timing issue while using the multiprocessing module.

Consider the following scenario. I have functions like this:

import multiprocessing as mp

def workerfunc(x):
    # timehook 3
    # something with x
    # timehook 4

def outer():

    # do something

    mygen = ... (some generator expression)

    pool = mp.Pool(processes=8)

    # time hook 1
    result = [pool.apply(workerfunc, args=(x,)) for x in mygen]
    # time hook 2

if __name__ == '__main__':
    outer()

I am utilizing the time module to get an arbitrary feeling for how long my functions run. I successfully create 8 separate processes, which terminate without error. The longest time for a worker to finish is about 130 ms (measured between timehook 3 and 4).

I expected (as they are running in parallel) that the time between hook 1 and 2 will be approximately the same. Surprisingly, I get 600 ms as a result.

My machine has 32 cores and should be able to handle this easily. Can anybody give me a hint where this difference in time comes from?

Thanks!

This depends on how many tasks do you have. You have 8 processes which means you can do 8 tasks at the same time. But if you have more than 8 tasks, it is obvious that you need more time right? For example if you have 9 tasks, using 8 processes should cost double time as one task needs. — Sraw, Apr 04 '19 at 16:18
@Sraw: that is true, but I have exactly 8 tasks to run. I manipulate 8 pandas DataFrame objects and write the result of into an array (result in my example above). — DocDriven, Apr 05 '19 at 06:58

Pitto · Answer 1 · 2019-04-05T07:31:49.610

Since you are using multiprocessing and not multithreading your performance issue is not related to GIL (Python's Global Interpreter Lock).

I've found an interesting link explaining this with an example, you can find it in the bottom of this answer.

The GIL does not prevent a process from running on a different processor of a machine. It simply only allows one thread to run at once within the interpreter.

So multiprocessing not multithreading will allow you to achieve true concurrency.

Lets understand this all through some benchmarking because only that will lead you to believe what is said above. And yes, that should be the way to learn — experience it rather than just read it or understand it. Because if you experienced something, no amount of argument can convince you for the opposing thoughts.

import random
from threading import Thread
from multiprocessing import Process
size = 10000000   # Number of random numbers to add to list
threads = 2 # Number of threads to create
my_list = []
for i in xrange(0,threads):
    my_list.append([])
def func(count, mylist):
    for i in range(count):
        mylist.append(random.random())
def multithreaded():
    jobs = []
    for i in xrange(0, threads):
        thread = Thread(target=func,args=(size,my_list[i]))
        jobs.append(thread)
    # Start the threads
    for j in jobs:
        j.start() 
    # Ensure all of the threads have finished
    for j in jobs:
        j.join()

def simple():
    for i in xrange(0, threads):
        func(size,my_list[i])

def multiprocessed():
    processes = []
    for i in xrange(0, threads):
        p = Process(target=func,args=(size,my_list[i]))
        processes.append(p)
    # Start the processes
    for p in processes:
        p.start()
    # Ensure all processes have finished execution
    for p in processes:
        p.join()
if __name__ == "__main__":
    multithreaded()
    #simple()
    #multiprocessed()

Additional information

Here you can find the source of this information and a more detailed technical explanation (bonus: there's also Guido Van Rossum quotes in it :) )

Thank you for your answer, I am aware of the GIL. Because of that, I thought using processes instead of threads would be the way to go. Did I miss something? — DocDriven, Apr 05 '19 at 07:06
Good point, @DocDriven! There's an example in the 1st link I provided that uses time to check execution times and compares multiprocessing / multithreading. Maybe that can be helpful to check if changes can be implemented in your code too. For instance he is using Process and not Pool. — Pitto, Apr 05 '19 at 07:16

Vince W. · Answer 2 · 2019-04-04T15:55:47.157

1

You are using pool.apply which is blocking. Use pool.apply_async instead and then the function calls will all run in parallel, and each will return an AsyncResult object immediately. You can use this object to check when the processes are done and then retrieve the results using this object also.

edited Apr 04 '19 at 15:55

answered Apr 04 '19 at 15:50

Vince W.

3,561
3
31
59

Thank you, this speeds up the process to an acceptable level. But I'm still a bit confused how these processes block each other. Could you explain this a little more, please? – DocDriven Apr 05 '19 at 07:16
1

"Pool.apply is like Python apply, except that the function call is performed in a separate process. Pool.apply blocks until the function is completed." complete post here: https://stackoverflow.com/questions/8533318/multiprocessing-pool-when-to-use-apply-apply-async-or-map – Pitto Apr 05 '19 at 07:32
Right, when you call apply, the main process says you go do this job and I’ll wait here until you’re done. Apply-async says you go do this job and give me a claim ticket and I’ll come back letter to pick up the result – Vince W. Apr 05 '19 at 10:49

Timing a multiprocessing script

2 Answers2