How to parse unstructured JSON file in Node?

Question

I have to parse a JSON file that has many objects but no structure to the file. It looks like this:

{"obj1": "john"}
{"obj2": "sally"}
{"obj3": "veronica"}

Each object is on on it's own there is no container. So when I open the file and try to iterate through it I get the error Unexpected token { in JSON

Aside from wrapping the objects in an array and then manually going through the whole file to add commas, how can I parse this?

Unfortunately, that's not json. so.... you can't parse it as if it were json. You could possibly iterate over the file, line by line, and then eval the objects, maybe — Kevin B, Mar 20 '17 at 18:40
Yeah, it's not that it's an unstructured JSON file, it's its own format. JSON requires double quotes (not single quotes) around keys and strings, a single top-level thing, and commas between array entries and object properties. — T.J. Crowder, Mar 20 '17 at 18:42
Sorry, it is in double quotes. Just a habit working in singles — Max Phillips, Mar 20 '17 at 18:47
Ah, then just [iterate line by line](http://stackoverflow.com/questions/6156501/read-a-file-one-line-at-a-time-in-node-js) and parse each one, adding it to an array. — Kevin B, Mar 20 '17 at 18:48

score 2 · Accepted Answer · edited May 23 '17 at 10:30

If it's really one-object-per-line, it's fairly straightforward to take the string, break it into lines, and JSON.parse each line:

const str =
  '{"obj1": "john"}\n' +
  '{"obj2": "sally"}\n' +
  '{"obj3": "veronica"}';

const array = str.split(/[\r\n]+/)
                 .map(entry => JSON.parse(entry));

console.log(array);

...but that's assuming it really is one object per line.

If you're reading the file, you don't have to start out with all in one string as above; just read line by line as Kevin B points out.

(Since you're using Node, I've happily used ES2015+ features above...)

score 1 · Answer 2 · edited Mar 20 '17 at 19:28

If you assume each line of the input file is complete, self-standing JSON, then a split-into-lines-then-parse-each strategy works well.

But even if the data isn't limited to a single line, not all is lost. You can heuristically parse the file. It isn't hyper-efficient, but except for very large files you'll probably never know the difference:

function incrementallyParseJSON(filepath) {
    var lines = fs.readFileSync(filepath)
                  .toString()
                  .split(/\n/g);
    var result = [];
    var [start, stop] = [0, 1];
    while (stop <= lines.length) {
        try { 
            var part = JSON.parse(lines.slice(start, stop).join('\n'));
            result.push(part);
            [start, stop] = [stop, stop+1]; 
        } catch (e) {
            stop += 1;
        }
    }
    return result;
}

So if your file is:

{"obj1": "john"}
{"obj2": "sally",
 "more": "other"}
{"obj3": "veronica"}
"something"
12

The result will be:

[ { obj1: 'john' },
  { obj2: 'sally', more: 'other' },
  { obj3: 'veronica' },
  'something',
  12 ]

Example:

function incrementallyParseJSON(str) {
    var lines = str.split(/\n/g);
    var result = [];
    var [start, stop] = [0, 1];
    while (stop <= lines.length) {
        try { 
            var part = JSON.parse(lines.slice(start, stop).join('\n'));
            result.push(part);
            [start, stop] = [stop, stop+1]; 
        } catch (e) {
            stop += 1;
        }
    }
    return result;
}
var str =
  '{"obj1": "john"}\n' +
  '{"obj2": "sally",\n' +
  ' "more": "other"}\n' +
  '{"obj3": "veronica"}\n' +
  '"something"\n' +
  '12';

console.log(incrementallyParseJSON(str));

How to parse unstructured JSON file in Node?

2 Answers2