The easiest way to write multiprocessing and multiprocessing

Question

To be short，I use python and I have two functions:

def func1(item):
    do something

def func2(item):
    do something
    return result

And I have a list list = [item1, item2, ...]. Each element is the argument of function func1 and func2. That is, I want to do the following two loops:

for item in list:
    func1(item)

result_list = [func2(item) for item in list]

Now I want to apply threading and multiprocessing on these two loops. So what is the easiest way (I mean, in just several lines of codes) to achieve it?

For multiprocessing on func1, I like the following code:

with multiprocessing.Pool(processes = 6) as p:
    p.imap(func1, list)

Do we have similar code for other cases? That is, threading on func1 and func2, multiprocessing on func2.

Not exactly. What about func2? I need to collect the returned result? — Joesf.Albert, Oct 25 '21 at 11:38
What about it? You collect the results in exactly the same way: multiprocessing.ThreadPool provides the same api, so you get the results the same you would with multiprocessing.Pool, i.e. either use `map` or iterate an `imap`. I might have misunderstood, however — 2e0byo, Oct 25 '21 at 11:41

Himanshu Pingulkar · Answer 1 · 2021-10-25T11:51:35.030

import multiprocessing
import threading

def func1(item):
    do something

def func2(item):
    do something
    return result


# Creating thread
t1 = threading.Thread(target=func1, args=(10,))
t2 = threading.Thread(target=func2, args=(10,))
# starting thread 
t1.start()
t2.start()
# wait until thread is completely executed
t1.join()
t2.join()

# Creating processes
p2 = multiprocessing.Process(target=func2, args=(10, ))
# Starting processes
p2.start()
# Wait until process is finished
p2.join()

https://www.geeksforgeeks.org/multiprocessing-python-set-1/

https://www.geeksforgeeks.org/multithreading-python-set-1/

OP specifically requests threading not multiprocessing on func1 (for some reason) — 2e0byo, Oct 25 '21 at 11:42

The easiest way to write multiprocessing and multiprocessing

1 Answers1