-1

I want to split a long text.csv text file of say 2 x 400,000 size into 80 smaller files (2 x 6000). Ideally, I want to write each smaller file on to my disk as text1.csv, text2.csv etc. What is the most efficient way to do this?

         col 1         col2
0        text          text
1        text          text
2        text          text
...
399999   text          text
400000   text          text

The smaller files would look like this:

         col 1         col2
0        text          text
1        text          text
2        text          text
...
4999     text          text
5000     text          text


         col 1         col2
5001     text          text
5002     text          text
5003     text          text
...
9999     text          text
10000    text          text

etc.
twhale
  • 725
  • 2
  • 9
  • 25
  • What have you tried to do? – dawg Oct 18 '18 at 13:33
  • I would not write anything for this myself, but use the split command on the shell - or possibly call that as an external program from python. See: https://stackoverflow.com/questions/2016894/how-to-split-a-large-text-file-into-smaller-files-with-equal-number-of-lines – rje Oct 18 '18 at 13:33
  • @rje: is it that simple? Can you post this as an answer so I can approve it? – twhale Oct 18 '18 at 13:37
  • @twhale done :) – rje Oct 18 '18 at 13:45

1 Answers1

0

I would not write anything for this myself, but use the split command on the shell - or possibly call that as an external program from python. See:

How to split a large text file into smaller files with equal number of lines?

rje
  • 6,388
  • 1
  • 21
  • 40