How to parse JSON that is larger than 256mb?

Asked Apr 21 '16 at 13:07

Active Apr 21 '16 at 14:13

Viewed 323 times

I'm migrating my app's backend from Parse to Apple's CloudKit. I have a file Datapoint.json that is nearly a gigabyte (represents 2.5 million Parse objects). I'm trying to import this in my migration.js script but I keep getting the error toString failed.

I read up on this error and it seems that a file can't be bigger than 256mb. What should I do?

var json = JSON.parse(fs.readFileSync('./Datapoint.json', 'utf8'));

edited Apr 21 '16 at 14:13

Jonas Köritz

2,606
21
33

asked Apr 21 '16 at 13:07

shakked

what operating system are you working with? – lipp Apr 21 '16 at 13:08
1

Use a streaming JSON parser. And don't try to `readFileSync`. – Bergi Apr 21 '16 at 13:09
OS X 10.11.4 my computer has a 3.1ghz i7 and 16gb of ram – shakked Apr 21 '16 at 13:09
What are you going to do with `json`? – Bergi Apr 21 '16 at 13:09
Did you use google? http://www.ngdata.com/parsing-a-large-json-file-efficiently-and-easily/ – evolutionxbox Apr 21 '16 at 13:09
I'm going to turn it into CloudKit records and save them slowly – shakked Apr 21 '16 at 13:10
http://stackoverflow.com/questions/29766868/node-js-read-big-file-with-fs-readfilesync – lipp Apr 21 '16 at 13:10
First read file asynchronously then use JSON Parser. – bilalmetla Apr 21 '16 at 13:11
2

@bilalmetla what's the advantage of readying it asynchronously? It's a one time operation – shakked Apr 21 '16 at 13:12
Use stream based solutions find out how the file is structured and use this stream library https://github.com/dominictarr/JSONStream. – Risto Novik Apr 21 '16 at 13:13
Your file is too big. This will block event loop. Asynchronoulsy it will work fine. – bilalmetla Apr 21 '16 at 13:16
@bilalmetla the JSON.parse is not async function so it will block the event loop, but if you split it into stream like I commented before it would be parse the chunks easily. – Risto Novik Apr 21 '16 at 13:18
To be fair, on a 64-bit OS it really *should* be possible to work with an in-memory gigabyte file. – Pointy Apr 21 '16 at 13:18
Yes you can use any external library for this. – bilalmetla Apr 21 '16 at 13:25
@bilalmetla I'm still running out of memory with JSONStream – shakked Apr 21 '16 at 13:31
make sure you understand the inside structure how the rows/objects locate, .pipe(JSONStream.parse('results.*')) – Risto Novik Apr 21 '16 at 13:34
Can you share exact error. May be your node process memeory exceeds. – bilalmetla Apr 21 '16 at 13:39
The parse backup structure should be like this http://blog.parse.com/learn/engineering/one-click-export/, so you should be able to use this ```cat backup.json | JSONStream 'results.*'``` – Risto Novik Apr 21 '16 at 13:43

How to parse JSON that is larger than 256mb?

0 Answers0