5

What is the best way to call same function in separate threads and have a separate list with returned values for each instance, without duplicating function?

Example:

import threading


def function(a):

    returned_values = []

    ct = threading.currentThread() 
    while getattr(ct, "do_run", True):
        ret = do_something(a)
        returned_values.append(ret)


t1 = threading.Thread(target=function, args=("AAA",))
t2 = threading.Thread(target=function, args=("BBB",))
t3 = threading.Thread(target=function, args=("CCC",))

t1.start()
t2.start()
t3.start()

import time;time.sleep(10)
t1.do_run = t2.do_run = t3.do_run = False

EDIT: Forgot to mention that I use Python 2.7

rbs
  • 169
  • 4
  • 10
  • Possible duplicate of [How to use threading in Python?](http://stackoverflow.com/questions/2846653/how-to-use-threading-in-python) – philshem Apr 27 '17 at 11:30

3 Answers3

7

Use ThreadPool

Something like this 

from multiprocessing.pool import ThreadPool

pool = ThreadPool()
pool.map(function, list_containing_args)

P.S it works similar to multiprocess map.Each argument is given a new thread .You can specify the number of threads you want to spawn if you have limited resources or a big list

from multiprocessing.pool import ThreadPool
import subprocess
def func(ip):
    c=subprocess.Popen("ping -c 3 "+ip, shell=True, stdout=subprocess.PIPE)
    output, error= c.communicate()
    return output

pool = ThreadPool()
for i in  pool.map(func,["127.0.0.1", "www.google.com", "www.facebook.com"]):
    print i
vks
  • 67,027
  • 10
  • 91
  • 124
  • Thanks. Could you please explain a bit how this works? – rbs Apr 26 '17 at 10:55
  • Is there a reason why I would use ThreadPool instead of multiprocessing.Pool class? – rbs Apr 26 '17 at 11:45
  • 1
    @rbs ThreadPool will launch `threads` which are relatively easy and cheap if you are doing `IO` related work.If you use `multiprocess` it will launch `Processes` which have overheads.So if your task involves `IO` use `threads` else `processes` – vks Apr 26 '17 at 11:47
  • I am not sure I understand. Using ThreadPool doesn't make the threads share the same memory space of the function? Isn't this going to impact how the separate calls to the functions work? And finally, how is this different then using threading? – rbs Apr 27 '17 at 09:40
  • @rbs ThreadPool is same as using threads.....Multiprocess.pool.map is different.I think my last comment confused you.Multiprocess is not multiple threads,its multple processes.But mutliprocess has one class Threadpool which uses threads instead of process. – vks Apr 27 '17 at 09:43
  • So where is the difference? Isn't the memory of the function going to be shared if using ThreadPools? Isn't this going to screw up what the function returns for each individual parallel call? – rbs Apr 27 '17 at 09:52
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/142816/discussion-between-rbs-and-vks). – rbs Apr 27 '17 at 09:56
  • @rbs see a working example....It will return a list of all responses... – vks Apr 27 '17 at 11:21
  • 1
    [Here's an explanation](http://stackoverflow.com/a/28463266/2327328) of how ThreadPool works – philshem Apr 27 '17 at 11:30
2

Wouldn't a ProcessPool here be better suited since threading is best suited for network I/O issues where a ProcessPool is best suited for memory intensive tasks.

from concurrent.futures import ProcessPoolExecutor

with futures.ProcessPoolExecutor(max_workers=n) as executor:
    executor.map(fn, args)
gold_cy
  • 13,648
  • 3
  • 23
  • 45
  • Why is this answer ignored? Is it a bad pattern, is there any issue in using this, or is it simply more common to use the pattern of the chosen answer? – Eduardo Pignatelli Mar 20 '18 at 16:14
  • @EduardoPignatelli this is not there in Python 2.7 – vks Oct 22 '19 at 09:27
  • @vks you can install it for Python 2.7 if needed so your statement is not exactly accurate – gold_cy Oct 22 '19 at 12:20
  • @aws_apprentice not entirely but true in default state....also person working on python 2.7 will prefer to work with python 2 features I assume – vks Oct 22 '19 at 12:38
0

If you insist on threading, you may do it like this:

  1. Set your arguments in advance

    n_thread, args_set = 3, [('AAA',), ('BBB',), ('CCC',)]

  2. Store all instances in a list

    threads = [threading.Thread(target=function, args=args_set[i])
           for i in range(n_thread)]
[t.start() for t in threads]

  3. Or use t1, t2, etc..

    for i in range(n_thread):
    var_thread = locals()['t%d' % i]
    var_thread = threading.Thread(target=function, args=args_set[i])
    var_thread.start()
print t1, t2
Roll
  • 289
  • 1
  • 5