0

I am trying to make an API validation from a given list of IP addresses using multiprocessing. All the processes contain the valid IPs will be stored in a shared queue. However, when I try to print the queue after the processes in the multiprocessing were executed, the final queue was empty. Could somebody show me how would I able to fix the code?

def validate(ip_address, results):
  try:
    response = requests.get('....API_URL....'))
    if response.status_code is 200:
        """ store in queue if valid """
        results.put(multiprocessing.current_process())

  except ConnectionError:
    pass

def main():
  """ ip_list is given """
  results = multiprocessing.Queue()

  processes = []
  for ip in ip_list:
      process = multiprocessing.Process(target = validate, args=(ip,results))
      processes.append(process)

  for p in processes:
      p.start()

  for p in processes:
      p.join()

  while results.empty() is False:
      result = results.get()
      final_list.append(result)

  print(final_list) ### EMPTY queue

main()

EDIT1: I tried to replace results.put(multiprocessing.current_process()) with something random such as results.put("randomTEXT") but final_list still prints empty. Also, could someone please suggest me a way to store the identity/name/id of the process that successfully requests the API?

סטנלי גרונן
  • 2,917
  • 23
  • 46
  • 68
curiousP
  • 69
  • 6
  • 1
    In `validate` did you try to replace the conditional statement and suite with a `put('somestring')` statement? did it work? Did you try to `put` something in the `except` suite? - were there exceptions? – wwii Jul 02 '19 at 21:13
  • 2
    `multiprocessing.current_process()` returns an instance of `multiprocessing.context.Process`. This kind of object can not be passed through a `mp.Queue` because "Pickling an AuthenticationString object is disallowed for security reasons". You can see this error message by enabling logging: `import logging`, `logger = mp.log_to_stderr(logging.INFO)`. – unutbu Jul 02 '19 at 21:16
  • Thanks for your help! So I tried to replace `results.put(multiprocessing.current_process())` with `results.put("randomTEXT")` but final_list still prints empty. Also, could you suggest me a way to store the identity/name/id of the process that successfully request the API? Many thanks! – curiousP Jul 02 '19 at 21:24
  • 1
    @curiousP: You could put the string `mp.current_process().name` in the queue. – unutbu Jul 02 '19 at 21:30
  • 1
    @curiousP: Building on wwii's idea, perhaps use `results.put((multiprocessing.current_process().name, response.status_code))` **outside** of the `if-statement` so you can see what `response.status_code`s you are getting. – unutbu Jul 02 '19 at 21:36
  • 1
    @curiousP: When comparing numbers, [use `==`, not `is`](https://stackoverflow.com/a/133024/190597). In your case, that means you should be using `if response.status_code == 200`. – unutbu Jul 02 '19 at 21:38
  • 1
    `current_process().name` works nicely! Thank you very much! – curiousP Jul 02 '19 at 21:40

0 Answers0