How do I use shell to split large file into pieces

Question

I have a very large file looks like this :

//abc/file1.js

some javascript code

//abc/file2.js

some javascript code

//abc/file3.js

some javascript code

Here I want to split this large file into pieces and store the pieces into file1.js,file2.js etc.

Answer 1

You can do this with awk. Print out each input line, but to a file name that changes whenever the input line indicates that a new file starts.

awk '
    /^\/\/abc\// { filename = $1; sub(/.*\//, "", filename); next; }
    filename { print >filename }
'

Remove the call to next if you want the header lines to be included, e.g. to have //abc/file1.js as the first line of file1.js. You may want to tweak the code that recognizes header lines depending on your requirements. Text prior to the first header line will not be printed anywhere; change filename { … } to 1 { … } if you want to print it to standard output.

Answer 2

Try csplit -k -f file - '/^\/\//' '{1000}' < largefile.

Adjust 1000 to a suitable number. If there are n files in largefile, use n-2 instead of 1000.

If you're using GNU csplit, you can simply use * instead of 1000.

If there are many files in largefile, you'll need also to use -n 4 or some higher value.

Answer 3

If you can edit the file, and you know exactly where you want to split, not by some byte offset, then just copy the new pieces to new files and save those new pieces and the existing files with the names you want. That is, using the editor itself.