Python - Write to new file every nth row

Question

I have a long list of subscribers. Right now I'm writing all of them to one csv file. I want to split them up so that every 1000th row I create a new file and start writing to that, and so on until finished.

Current code:

with open(path, mode='w', newline='') as filename:
        writer = csv.writer(
            filename,
            delimiter=';',
            quotechar='"',
            quoting=csv.QUOTE_MINIMAL
        )

        writer.writerow(['email', 'name'])

        for subscriber in subscribers:
            writer.writerow([subscriber['Email'], subscriber['Name']])

How can I extend this to get the functionaly I need?

Make simple counter. If counter > 1000: then create new file and reset. — Erik Šťastný, Jul 17 '19 at 09:51

score 2 · Accepted Answer · answered Jul 17 '19 at 09:55

you can chunk your subscribers list:

def chunks(l, n):
    """Yield successive n-sized chunks from l."""
    for i in xrange(0, len(l), n):
        yield l[i:i + n]

for subscribers_thousand in chunks(subscribers, 1000):
    # Remember to change path for each 1000 subscribers chunk
    with open(path, mode='w', newline='') as filename:
        writer = csv.writer(
            filename,
            delimiter=';',
            quotechar='"',
            quoting=csv.QUOTE_MINIMAL
        )
        for subscriber in subscribers_thousand:
            writer.writerow([subscriber['Email'], subscriber['Name']])

Remember to change file name in each iteration so you do not overwrite your files.

NOTE: I got this chunk method from this topic in SO. It works like a charm

looks so handy and elegant! – Swadhikar Jul 17 '19 at 10:09 — Swadhikar, Jul 17 '19 at 10:09

Python - Write to new file every nth row

1 Answers1