How to load JSON data on Node.js

Question

I am working to load JSON data using Node/express.js and plot it on map. As first start, I am inspired by the example presented in the repp leaflet-geojson-stream. https://github.com/tmcw/leaflet-geojson-stream/tree/master/example

Client: https://github.com/tmcw/leaflet-geojson-stream/blob/master/example/client.js

var L = require('leaflet'),
    leafletStream = require('../');

L.Icon.Default.imagePath = 'http://leafletjs.com/dist/images';

window.onload = function() {
    var div = document.body.appendChild(document.createElement('div'));
    div.style.cssText = 'height:500px;';
    var map = L.map(div).setView([0, 0], 2);
    L.tileLayer('http://a.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);
    var gj = L.geoJson().addTo(map);
    leafletStream.ajax('/points.geojson', gj)
        .on('end', function() {
        });
};

Server : https://github.com/tmcw/leaflet-geojson-stream/blob/master/example/server.js

var express = require('express'),
    browserify = require('browserify'),
    app = express();

app.get('/', function(req, res) {
    res.send('<html><head><link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.6.4/leaflet.css" /></head><body><script src="bundle.js"></script></html>');
});

app.get('/bundle.js', function(req, res) {
    var b = browserify();
    b.add('./client.js');
    b.bundle().pipe(res);
});

app.get('/points.geojson', function(req, res) {
    res.write('{"type":"FeatureCollection","features":[');
    var i = 0, die = 0;
    function send() {
        if (++i > 20) {
            res.write(JSON.stringify(randomFeature()) + '\n,\n');
            i = 0;
        } else {
            // it seems like writing newlines here causes the buffer to
            // flush
            res.write('\n');
        }
        if (die++ > 1000) {
            res.write(JSON.stringify(randomFeature()));
            res.write(']');
            res.end();
            return;
        }
        setTimeout(send, 10);
    }
    send();
});

app.listen(3000);

function randomFeature() {
    return {
        type: 'Feature',
        geometry: {
            type: 'Point',
            coordinates: [
                (Math.random() - 0.5) * 360,
                (Math.random() - 0.5) * 180
            ]
        },
        properties: {}
    };
}

In the project, they create random json file. I wanted to read json file, then plot it. The reason I want to "Stream data" is to deal with the size of file (I know that there is better and easier ways to load json data), But I wanted to use this module.

I modified the server script :

var express = require('express'),
    browserify = require('browserify'),
    app = express();

app.get('/', function(req, res) {
    res.send('<html><head><link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.6.4/leaflet.css" /></head><body><script src="bundle.js"></script></html>');
});

app.get('/bundle.js', function(req, res) {
    var b = browserify();
    b.add('./client.js');
    b.bundle().pipe(res);
});

var o = require('../output_.geojson');

app.get('/points.geojson', function(req, res) {
         res.json(o);
});

app.listen(3000);

    res.write('');

But I am getting error :

/Users/macbook/leaflet-geojson-stream/output_.geojson:1
(function (exports, require, module, __filename, __dirname) { "type":"FeatureC
                                                                    ^
SyntaxError: Unexpected token :
    at Module._compile (module.js:439:25)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/Users/macbook/leaflet-geojson-stream/example/server.js:15:9)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)

Can anyone give a hint on what should I do in order to read and load json external file to plot the data.

score 2 · Answer 1 · edited May 23 '17 at 12:20

2

The : isn't expected, because you're in a function at that point.

To parse JSON, you simply have to call JSON.parse(), as stated in How to parse JSON using Node.js?. So you read the file, get the JSON String that's in there, and put it through JSON.parse()

edited May 23 '17 at 12:20

Community

1
1

answered Mar 31 '14 at 12:53

weeknie

160
7

I am using express.js so I am getting the file through this function : app.get('/points.geojson', function(req, res) { res.json(o); }) – user3378649 Mar 31 '14 at 12:55
Then just call JSON.parse() on the returned string;) – weeknie Mar 31 '14 at 12:57
Then what do you get as a return value exactly? can you copy it? – weeknie Mar 31 '14 at 13:58
I am getting the same error message mentioned above – user3378649 Mar 31 '14 at 14:20
As said before, the : isn't expected there because at that point you're defining a function, not an object. That's simply a syntax error, it doesn't have to do with the JSON problem – weeknie Mar 31 '14 at 16:17

score 1 · Answer 2 · answered Mar 31 '14 at 15:10

1

You are currently loading the whole JSON file into memory by 'requiring' it.

Instead you want to stream the file because it is big and so use the fs.createReadStream function:

var fs = require('fs');

app.get('/points.geojson', function(req, res) {
  res.setHeader('Content-Type', 'application/json');
  fs.createReadStream(__dirname + '../output_.geojson').pipe(res);
});

Also make sure that the contents of ../output_.geojson is actually valid JSON. You can use JSONLint to check - the file should with '{' or '[' and NOT have Javascript functions inside.

answered Mar 31 '14 at 15:10

Bino Carlos

1,357
8
10

But if you're going to stream the file, how will you convert it into JSON? I haven't heard of any function that will do this, not in standard javascript anyway – weeknie Mar 31 '14 at 16:19
1

The point is to get the contents of the JSON into the browser and so you don't need to convert it. JSON is text. Only once you have parsed the text does it become an object in memory. You want the object in memory in your browser. Therefore parsing the JSON into an object on the server is not a good plan :) – Bino Carlos Mar 31 '14 at 16:30
You could just as well decide to convert the JSON into an object and plot it there, sending only the graph to the client, since you don't know how much memory the client has available for you, but the server might. Sending a huge amount of data to the client is, in my opinion, a lot worse than putting it temporarily into the server memroy. – weeknie Mar 31 '14 at 16:34
Then we are in the land of filtering the stream somehow. You are right about not dumping the whole data to the browser but the same problem multiplies a thousand times on a busy server. – Bino Carlos Mar 31 '14 at 16:35
@Bino Carlos: If I understand you very well. This can solve the problem but it's not a good plan – user3378649 Mar 31 '14 at 16:53

How to load JSON data on Node.js

2 Answers2

Linked