How to get data out of a Node.js http get request

Question

I'm trying to get my function to return the http get request, however, whatever I do it seems to get lost in the ?scope?. I'm quit new to Node.js so any help would be appreciated

function getData(){
  var http = require('http');
  var str = '';

  var options = {
        host: 'www.random.org',
        path: '/integers/?num=1&min=1&max=10&col=1&base=10&format=plain&rnd=new'
  };

  callback = function(response) {

        response.on('data', function (chunk) {
              str += chunk;
        });

        response.on('end', function () {
              console.log(str);
        });

        //return str;
  }

  var req = http.request(options, callback).end();

  // These just return undefined and empty
  console.log(req.data);
  console.log(str);
}

Denys Séguret · Accepted Answer · 2017-08-23T05:55:17.517

134

Of course your logs return undefined : you log before the request is done. The problem isn't scope but asynchronicity.

http.request is asynchronous, that's why it takes a callback as parameter. Do what you have to do in the callback (the one you pass to response.end):

callback = function(response) {

  response.on('data', function (chunk) {
    str += chunk;
  });

  response.on('end', function () {
    console.log(req.data);
    console.log(str);
    // your code here if you want to use the results !
  });
}

var req = http.request(options, callback).end();

edited Aug 23 '17 at 05:55

answered Oct 23 '13 at 10:45

Denys Séguret

372,613
87
782
758

12

I would recommend pushing the chunks into an array and then use join('') in the end. That will avoid issues if there is lots of data – Eric Jun 04 '14 at 16:05
How do I get the HTTP response code of the response (200 or 404 etc.)? Is there any documentation about the keyword 'on' (response.on), 'data' and 'end'? Are these keywords? There seems to be nothing here: https://nodejs.org/api/http.html#http_class_http_serverresponse – Tyler Durden Aug 07 '17 at 21:47
@TylerDurden `statusCode` is a property of the response object. I couldn't find proper documentation for the `ServerResponse` object either, just examples in the docs for the `get` and `request` methods. – Phocacius Oct 15 '17 at 20:51
1

But this makes the code messy! Why is javascript designed like this? – Daniel Jan 15 '18 at 15:07
@Daniel There are now facilities to handle the asynchronous event model: Promises and async/await. – Denys Séguret Jan 15 '18 at 16:04
@DenysSéguret Would using a promise library (ie Axios) correctly concatenate the returned data then? So can I assume the .then with return all the data and the .catch will return a complete error if any)? – Matthew Mullin May 14 '19 at 08:37

score 33 · Answer 2 · answered Jun 12 '19 at 15:52

Simple Working Example of Http request using node.

const http = require('https')

httprequest().then((data) => {
        const response = {
            statusCode: 200,
            body: JSON.stringify(data),
        };
    return response;
});
function httprequest() {
     return new Promise((resolve, reject) => {
        const options = {
            host: 'jsonplaceholder.typicode.com',
            path: '/todos',
            port: 443,
            method: 'GET'
        };
        const req = http.request(options, (res) => {
          if (res.statusCode < 200 || res.statusCode >= 300) {
                return reject(new Error('statusCode=' + res.statusCode));
            }
            var body = [];
            res.on('data', function(chunk) {
                body.push(chunk);
            });
            res.on('end', function() {
                try {
                    body = JSON.parse(Buffer.concat(body).toString());
                } catch(e) {
                    reject(e);
                }
                resolve(body);
            });
        });
        req.on('error', (e) => {
          reject(e.message);
        });
        // send the request
       req.end();
    });
}

Thank for share!! This was the unique sample with return data without use console.log. — Altieres de Matos, Apr 30 '20 at 13:02
Seconded re @AltieresdeMatos's comment. This is a good, complete, *practical* example, that actually answers the original question as stated. This should be the accepted answer (in 2021) I reckon. Thanks pal. — Adam Cameron, Jan 19 '21 at 13:17

score 22 · Answer 3 · answered Oct 15 '16 at 07:00

22

Shorter example using http.get:

require('http').get('http://httpbin.org/ip', (res) => {
    res.setEncoding('utf8');
    res.on('data', function (body) {
        console.log(body);
    });
});

answered Oct 15 '16 at 07:00

Oded Breiner

28,523
10
105
71

1

This example is about as short as you can get and still run. This assumes a good url and small response. I prefer the http examples that chunk the data response, use the response `end` event, and use the request `errror` event. – dturvene Jun 26 '20 at 14:11
2

This answer is out of context given the question asked. Beside that, you are not explicitly listening for the `error event ` which will be triggered if the connection is lost while the request is in-progress or if any other issues occur during transmission. – rags2riches-prog Jul 18 '20 at 08:29
1

Warning: I think 'data' can give partial updates, the 'response' event is more practical. – moltenform Jan 28 '21 at 02:26

score 12 · Answer 4 · answered Dec 16 '15 at 20:41

12

from learnyounode:

var http = require('http')  

http.get(options, function (response) {  
  response.setEncoding('utf8')  
  response.on('data', console.log)  
  response.on('error', console.error)  
})

'options' is the host/path variable

answered Dec 16 '15 at 20:41

ezChx

4,024
1
19
16

thanks, been looking all over for a simple example and every one I found threw half a dozen new concepts at me. This just laid out how the http.get() works nice and simple. Excellent! – NickW Apr 30 '20 at 14:26
1

Warning: I think 'data' can give partial updates, the 'response' event is more practical – moltenform Jan 28 '21 at 02:27

score 9 · Answer 5 · edited Jul 13 '19 at 19:15

9

from learnyounode:

var http = require('http')
var bl = require('bl')

http.get(process.argv[2], function (response) {
    response.pipe(bl(function (err, data) {
        if (err)
            return console.error(err)
        data = data.toString()
        console.log(data)
    }))
})

edited Jul 13 '19 at 19:15

Trevor Sears

459
8
24

answered Mar 28 '16 at 05:45

Elise Chant

5,048
3
29
36

Vaibhav Rai · Answer 6 · 2019-06-14T09:39:31.807

I think it's too late to answer this question but I faced the same problem recently my use case was to call the paginated JSON API and get all the data from each pagination and append it to a single array.

const https = require('https');
const apiUrl = "https://example.com/api/movies/search/?Title=";
let finaldata = [];
let someCallBack = function(data){
  finaldata.push(...data);
  console.log(finaldata);
};
const getData = function (substr, pageNo=1, someCallBack) {

  let actualUrl = apiUrl + `${substr}&page=${pageNo}`;
  let mydata = []
  https.get(actualUrl, (resp) => {
    let data = '';
    resp.on('data', (chunk) => {
        data += chunk;
    });
    resp.on('end', async () => {
        if (JSON.parse(data).total_pages!==null){
          pageNo+=1;
          somCallBack(JSON.parse(data).data);
          await getData(substr, pageNo, someCallBack);
        }
    });
  }).on("error", (err) => {
      console.log("Error: " + err.message);
  });
}

getData("spiderman", pageNo=1, someCallBack);

Like @ackuser mentioned we can use other module but In my use case I had to use the node https. Hoping this will help others.

Thanks dude, I ran into the same situation like you, it sucks we can only use the `https` for issuing https request. — kenshinji, Oct 19 '19 at 07:41

score 3 · Answer 7 · answered Mar 16 '16 at 18:56

This is my solution, although for sure you can use a lot of modules that give you the object as a promise or similar. Anyway, you were missing another callback

function getData(callbackData){
  var http = require('http');
  var str = '';

  var options = {
        host: 'www.random.org',
        path: '/integers/?num=1&min=1&max=10&col=1&base=10&format=plain&rnd=new'
  };

  callback = function(response) {

        response.on('data', function (chunk) {
              str += chunk;
        });

        response.on('end', function () {
              console.log(str);
          callbackData(str);
        });

        //return str;
  }

  var req = http.request(options, callback).end();

  // These just return undefined and empty
  console.log(req.data);
  console.log(str);
}

somewhere else

getData(function(data){
// YOUR CODE HERE!!!
})

this seems not to work as callbackData() is not defined as a function? — Hugo Koopmans, Jul 14 '17 at 13:41

How to get data out of a Node.js http get request

7 Answers7

Linked