0

I'm new to multithreading with python and I've just tried the following solution. The problem is that the output file is empty each time and I can't find the problem :

from concurrent.futures import ThreadPoolExecutor
import threading
csv_writer_lock = threading.Lock()

def get_data(searchKey):
  #do some data scraping
  l1.append(data)
  l2.append(data)
  l = l1 + l2
  with csv_writer_lock:
    with open("results.csv", mode="a") as f1:
      writer = csv.writer(f1, delimiter=",")
      writer.writerows(l)
    f1.close()
  return l

def set_up_threads(urls):
  try:
    
    with ThreadPoolExecutor(max_workers=5) as executor:
        result = executor.map(get_data, urls["links"],timeout = 300)
  except Exception:
      print("Error")
  return result


df = pd.read_csv(file_path)
final = set_up_threads(df)

Somebody can point me in the direction of the error in this solution? thank you!

Seifeur
  • 41
  • 4
  • 1
    Possibly a duplicate of https://stackoverflow.com/questions/3976711/csvwriter-not-saving-data-to-file-the-moment-i-write-it ? (tl;dr make sure to call `flush` on the CSVWriter) – dtanabe Feb 24 '21 at 00:17
  • 3
    not sure why you're calling `f1.close()` since it's already called when you exit the `with` clause, other than that i see no reason for the csv to be empty – AntiMatterDynamite Feb 24 '21 at 00:47
  • If you do a `print(l)` before trying to write it, does it contain any data? :) – kamion Feb 24 '21 at 02:35

1 Answers1

0

I was able to resolve it using @dtanabe trick, i have to add flush.

Seifeur
  • 41
  • 4