0

am having the below mentioned code that will process large file with N lines at a time. But i need help related to processing of file, say if i want to start processing from line number: 121495, and want to process in batches of 10000.

from googletrans import Translator
from itertools import islice
file_translate = Translator()

with open("Ruprechter_at_data_27072020_de.csv", encoding="utf8") as file:
   while True:
      lines = list(islice(file, 10000))
      #print(lines)
      with open("test_04.csv",'a+',encoding="utf8") as file1:
            for line in lines:
                    #print(line)
                    x=file_translate.translate(line, dest='en')
                    file1.write(x.text+'\n')
      if not lines:
        break
MisterMiyagi
  • 44,374
  • 10
  • 104
  • 119
swagato
  • 3
  • 2
  • What have you tried so far? ``islice`` takes an argument for *start* and end, did you try that? What specific technical problem did you encounter trying to solve this? – MisterMiyagi Sep 05 '20 at 12:33
  • have tried with : lines = list(islice(file, 121495,None)), but it will read the complete file from that line number onwards and process it, it wont happen in batches – swagato Sep 05 '20 at 12:41
  • So if you have managed to set the start already, your question is *only* how to iterate in batches? – MisterMiyagi Sep 05 '20 at 12:46
  • yes, just need help to process in batches and keep iterating – swagato Sep 05 '20 at 12:50
  • Does this answer your question? [Iterate an iterator by chunks (of n) in Python?](https://stackoverflow.com/questions/8991506/iterate-an-iterator-by-chunks-of-n-in-python) – MisterMiyagi Sep 05 '20 at 13:10
  • Does this answer your question? [how to split an iterable in constant-size chunks](https://stackoverflow.com/questions/8290397/how-to-split-an-iterable-in-constant-size-chunks) – MisterMiyagi Sep 05 '20 at 13:11

0 Answers0