0

I am running a multiprocessing task in python, how can I timeout a function after 60seconds.

What I have done is shown in the snippet below:

import multiprocessing as mp
from multiprocessing import Pool
from multiprocessing import Queue


def main():
  global f
  global question
  global queue
  queue = Queue()   
  processes = []
  question = [16,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,17,18,19,21,20,23]
  cores=5
  loww=0
  chunksize = int((len(question)-loww)/cores)
  splits = []
  for i in range(cores):
    splits.append(loww+1+((i)*chunksize))
  splits.append(len(question)+1)
  print("",splits)
  args = []
  for i in range(cores):
    a=[]
    arguments = (i, splits[i], splits[i+1])
    a.append(arguments)
    args.append(a)
  print(args)
  p = Pool(cores)
  p.map(call_process, args)
  p.close()
  p.join

def call_process(args):
  ## end this whole block if it is taking more than 1 minutes
  starttime = time.time()
  lower=args[0][1]
  upper=args[0][2]
  for x in range(lower,upper):
    if time.time() >= starttime + 60: break
    a = question[x-1]
    try:
      pass
      # a lot of functions is called and returned here 
    except:
      continue
    #write item to file
    print('a = ',a)
   
  return a  


main()

I want to ensure that the call_process() method does not run for more than a minute for a particular value. Currently, I am using if time.time() >= starttime + 60: break which would not work effectively as I have different functions and things happening in the try and except block. What can I do?

Jesujoba Oluwadara ALABI
  • 499
  • 5
  • 16
  • 1
    Does any of the solutions in [this](https://stackoverflow.com/questions/26063877/python-multiprocessing-module-join-processes-with-timeout) post help you? – Thymen Dec 04 '20 at 14:48
  • Not fully, my function returns a value if it completes. I need to have those values. Also, I need an individual timeout for each process. – Jesujoba Oluwadara ALABI Dec 04 '20 at 18:21
  • The second answer in that post from [Raiden Drake](https://stackoverflow.com/a/29378611/10961342) indicates how you can put a timeout per process. What should be the return value when a process is timed out? – Thymen Dec 04 '20 at 21:44
  • Yes, the timeout appears to be general. I want each data item to be given equal time to be processed (60s). The one from Raiden Drake gives time to the whole worker. Which implies some data points will not be processed at all. Note that the worker is to process a batch of data and I want all the data point in the batch to be processed (either to completion or timeout) – Jesujoba Oluwadara ALABI Dec 05 '20 at 02:12

0 Answers0