Python multicore programming

Question

Please consider a class as follow:

class Foo:
    def __init__(self, data):
        self.data = data

    def do_task(self):
        #do something with data

In my application I've a list containing several instances of Foo class. The aim is to execute do_task for all Foo objects. A first implementation is simply:

 #execute tasks of all Foo Object instantiated
 for f_obj in my_foo_obj_list:
     f_obj.do_task()

I'd like to take advantage of multi-core architecture sharing the for cycle between 4 CPUs of my machine.

What's the best way to do it?

You can use multiprocessing module. https://docs.python.org/2/library/multiprocessing.html — NorthCat, May 08 '14 at 08:54
try [here](http://stackoverflow.com/a/1182350/1982962) it will help you — Kobi K, May 08 '14 at 08:56

score 22 · Answer 1 · answered May 08 '14 at 09:03

22

You can use process pools in multiprocessing module.

def work(foo):
    foo.do_task()

from multiprocessing import Pool

pool = Pool()
pool.map(work, my_foo_obj_list)
pool.close()
pool.join()

answered May 08 '14 at 09:03

timrau

22,578
4
51
64

+1 This is a very nice and fast to implement. – Kobi K May 08 '14 at 09:10
how do I wait so all processes would finish? – Flash Thunder Mar 13 '19 at 16:47
2

@FlashThunder When `pool.join()` returns all processes finished. – timrau Mar 14 '19 at 14:32

score 12 · Answer 2 · edited Nov 08 '17 at 01:59

Instead of going through all the multithreading/multicore basics, I would like to reference a Post by Ryan W. Smith: Multi-Core and Distributed Programming in Python

He will go into details how you can utilize multiple cores and use those concepts. But please be careful with that stuff if you are not familiar with general multithreading concepts.

Functional Programming will also allow you to customize the algorithm/function for each core.

Python multicore programming

2 Answers2

Linked