Write to different file every Nth iteration?

Question

I am trying to write to a different file every N iterations in a loop, and store everything that happened between that batch. I can accomplish this by using lists. For example,

import os

def write_to_file(data, fileName):
    file = os.path.join('/home/user/data', fileName)
    with open(file, 'a') as f:
        f.write(str(data))

TEMP = [] 
for i in range(50):
    TEMP.append(i)
    if i != 0 and i % 10 == 0:
        write_to_file(TEMP, 'data{}.txt'.format(i))
        TEMP = []

This will effectively write into a different file every 10th iteration , as expected, like the following:

File #1: [0, ... , 10]
File #2: [11, ..., 20]
...

But is there any other way of doing this without having to use a list? I am not having performance issues or anything but I feel like there most be another way of doing this without having to explicitly call a list.

Sure sounds like [premature optimzation](https://en.wikipedia.org/wiki/Program_optimization#When_to_optimize) to me. — martineau, Jun 04 '18 at 18:12

score 1 · Answer 1 · answered Jun 04 '18 at 18:05

If you don't want to temporarily store the work, your only option is to write to the file incrementally, like this:

nfilenum=0
newfileevery=7

fd = None
try: 
   for i in xrange(50):

      if (i%newfileevery)==0
         if fd is not None: fd.close()
         fd = open("{}.txt".format(nfilenum), 'w')

      fd.write("{} ".format(i)) # The stuff

finally:
   if fd is not None: fd.close()

score 0 · Answer 2 · answered Jun 04 '18 at 18:09

You can use the grouper recipe from itertools to accomplish your task:

from itertools import zip_longest


def grouper(iterable, n, fillvalue=None):
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)

for g in grouper(range(1, 51), 10):
    write_to_file(g, 'data{}.txt'.format(g[-1]))

score 0 · Answer 3 · answered Jun 04 '18 at 18:13

You can use this chunking recipe to write a list each time. In addition, you can enumerate to extract an index for each chunk. For example:

def chunks(l, n):
    """Yield successive n-sized chunks from l."""
    for i in range(0, len(l), n):
        yield l[i:i + n]

def write_to_file(data, fileName):
    file = os.path.join('/home/user/data', fileName)
    with open(file, 'a') as f:
        f.writelines(', '.join(map(str, data))+'\n')

for i, chunk in enumerate(chunks(list(range(1, 51)), 10)):
    write_to_file(chunk, 'data{}.txt'.format(i))

Write to different file every Nth iteration?

3 Answers3