I've got a list of links in my program, and I want to visit these links and collect some data from them, also I've got three other empty lists which I will append to based on the result. I don't know how to use multiprocessing on this task.
- All processes should read from the main list and will write(append) to one of other three lists. 
- How to distribute links between processes. 
- Is multiprocessing even suitable for this task? If yes witch method should be used? - def remove_redirect(): main_list = get_data_from_db() empty_list_1 = [] empty_list_2 = [] empty_list_3 = [] for link in main_list: req_result = manage_requests(link) if isinstance(req_result, bool): pass else: try: if req_result.find('div', class_="errorName"): empty_list_1.append(link) elif req_result.find('section', class_='filter'): empty_list_2.append(link) elif req_result.find('div', class_="proStatus"): empty_list_3.append(link) except Exception as error: print(error) pass if __name__ == "__main__": processes = [] for i in range(os.cpu_count()): print('Registering process %d' % i) processes.append(Process(target=remove_redirect)) for process in processes: process.start() for process in processes: process.join()
