0

I want to read a big data file 100 by 100. But my script reads only first 100 line, then it never enters in "for loop" for the rest. More clearly, in the end, I get only lines from 0 to 100, but I want also 100-200, 200-300, 300-400 etc. I can't see what the problem is.

(I use python 3.4. All data in .csv file is number. I already know to use islice() but I want particularly use enumerate() function.)

I would be very happy if you could help.

pathInput = "input.csv"
f = open(pathInput, 'r')
sizeOfList = 100
iD = 0
while iD<1000:
    dataset = []
    for i, line in enumerate(f):
       if i<(iD + sizeOfList):
           dataset.append(line)
    print(dataset)
    iD += sizeOfWindow
R.yan
  • 2,214
  • 1
  • 16
  • 33
Mas A
  • 207
  • 1
  • 11

1 Answers1

0

Basically, files objects are not a list, but it has a cursor that gets moved towards the end of the file as read is called (which enumerate basically does). The cursor basically is placed at the end of the file and thus nothing can be read anymore on the second loop.

Two ways this can be solved:

1) Read the whole file into memory first, and cast the enumerate call into a list, e.g.:

pathInput = "input.csv"
f = open(pathInput, 'r')
idx_line = enumerate(f)
id = 0
while id < 1000:
    dataset = []
    for idx, line in items:
        ...

2) Call seek on the file object to get back to the beginning of the file before calling enumerate(f).

while id < 1000:
    f.seek(0)
    dataset = []

Also refer to:

However, given that you probably have a CSV file, you probably want to try the following:

metatoaster
  • 17,419
  • 5
  • 55
  • 66