0

i have a loop for 2001 files:

for i in tqdm(result_file_list):
    df = read_pickle(file_path, i)
    final_df += df

and i want to save final_df every 100 files, whats the easiest way yo do it?

i have in mind only smth like this:

for i in tqdm(result_file_list[:100]):
    df = read_pickle(file_path, i)
    final_df += df 

for i in tqdm(result_file_list[100:200]):
    df = read_pickle(file_path, i)
    final_df += df
GrandConyon
  • 49
  • 6
  • Is `final_df` cumulative or a new variable every 100 files? – Kent Shikama Feb 17 '20 at 12:37
  • Does this answer your question? [Iterate an iterator by chunks (of n) in Python?](https://stackoverflow.com/questions/8991506/iterate-an-iterator-by-chunks-of-n-in-python) – mkrieger1 Feb 17 '20 at 12:45

1 Answers1

1

This should help, calculate how many iterations (your stages)

import math

#2001 / 100 = 20.01 (21 iteration, round it up 20.01 > 21)

stages = math.ceil(2001/100.0)

#for z in xrange(stages): #python2
for z in range(stages): #python3
    for i in tqdm(result_file_list[i*x: (i*z)+100]):
        your_code()
Wonka
  • 1,548
  • 1
  • 13
  • 20