1

I am learning the basics of multiprocessing and want to divide for loops into seperate processes to be processed on different cpu cores. I am planning to implement multiprocessing in an existing script, that does heavy computation.

Right now my approach looks like this:

import multiprocessing as mp
from functools import partial

def function(a, b, numbers):
    return a*b

if __name__ == '__main__':
    a = 1
    b = 2
    numbers = range(1000)
    func_part = partial(function, a, b)
    # Pool function
    result = mp.Pool().map(func_part, numbers)
    print(result)

    # equivalent for loop
    L = []
    for i in numbers:
        L.append(function(a, b, i))
    print(L)

  1. Is there a better approach in doing this?

  2. Is it possible to get the iterator numbers as first parameter of the function without breaking it? numbers has to be passed lastly by the map function it seems.

Artur Müller Romanov
  • 4,417
  • 10
  • 73
  • 132
  • How do you define "better"? – martineau Apr 08 '20 at 17:13
  • Efficiency and syntax simplicity? – Artur Müller Romanov Apr 08 '20 at 17:14
  • IMO that's too vague. – martineau Apr 08 '20 at 17:18
  • How about `more pythonic`? – Artur Müller Romanov Apr 08 '20 at 17:22
  • @ArturMüllerRomanov if you look in the docs [here](https://docs.python.org/3.7/library/multiprocessing.html#module-multiprocessing.pool) you will see an example of how to use `multiprocessing.Pool`, in the example the context manager it is used, I guess this can be considered a pythonic example – kederrac Apr 08 '20 at 17:30
  • 1
    @kederrac's suggestion of using the `Pool` as context manager fits that description. If you don't care about order, using `map_sync()` might be more efficient in the sense of allowing more concurrent processing to take place (since the processes don't depend on each other). – martineau Apr 08 '20 at 17:30
  • @martineau Doesn't `map_async()` respect the order, just like `map()`? I believe `imap_unordered()` is the method to use if you don't care about the order. – AMC Apr 08 '20 at 18:51

1 Answers1

1

you can use a context manager:

with mp.Pool() as p:
    print(p.map(func_part, numbers))

Is it possible to get the iterator numbers as first parameter of the function without breaking it?

yes, it is possible, but you are not passing the iterator to the function, you are passing an item from your numabers variable

def function(number, a, b, ):
    return a*b

a = 1
b = 2
numbers = range(1000)
func_part = partial(function, a=a, b=b)



with mp.Pool() as p:
    print(p.map(func_part, numbers))
kederrac
  • 16,819
  • 6
  • 32
  • 55