NodeJS http.request not returning data even after specifying return on the 'end' event

Question

Basically I am trying to scrap some data from website and perform the DOM extraction, deletion and updation on a callback function binded to the 'end' event of http.request.

I have returned the data from the 'end' event callback too but it is not receiving in my route callback function. I get undefined there.

Below is the code block:

var scraper = {
    extractEmail: function (directoryName) {
        var result = getDirectory(directoryName);
        if (result !== 404) {
            var protocol = result.https ? https : http;
            protocol.request({
                host: 'somevalue.net',
                method: "GET"
            }, function (res) {
                var data = '';
                res.on('data', function (chunk) {
                    data += chunk;
                });

                res.on('end', function () {
                    return data;
                });
            })
                .on('error', function (err) {
                    return err;
                })
                .end();
            //return data;
        }
        else {
            //return "Failed";
        }
    }
};

And here is the Routes.js function:

app.get('/:directory', function (req, res) {
    var n = scraper.extractEmail(req.params.directory);
    console.log(n);
    res.send(n);
});

In here also I don't get the value of n.

score 1 · Answer 1 · answered Jan 19 '17 at 14:56

Is your 'var scraper' also in the route.js file? I guess it's not and you are unable to access that other js file, for doing so use module.exports.

eg.

// module.js
var name = "foobar";
// export it
exports.name = name; 



Then, in route.js...
>      //route.js
>      // get a reference to your required module
>      var myModule = require('./module'); 
>      //correct path to folder where your above file is
>      // name is a member of myModule due to the export above
>      var name = myModule.name;

I have exported it as well.. My full block of code looks like this: `var scraper = {};` `module.exports = scraper` .. But it is not working. — ChanX, Jan 20 '17 at 05:04

score 1 · Answer 2 · edited May 23 '17 at 12:24

You cannot return a value from an asynchronous callback. Well, you can, but it most likely will get ignored and most certainly won't do what you want.

You cannot even return a promise in that place. You can only resolve a promise where you now use the return statements. You need to return a promise from the main function and then resolve or reject the promise in your event handlers where you use returns now.

For more info see those answers:

score 0 · Answer 3 · answered Jan 19 '17 at 15:07

The simplest modification that will make this work is to pass a callback function to extractEmail to receive the data once it's ready.

var scraper = {
    extractEmail: function (directoryName, cb) {
        var result = getDirectory(directoryName);
        if (result !== 404) {
            var protocol = result.https ? https : http;
            protocol.request({
                host: 'somevalue.net',
                method: "GET"
            }, function (res) {
                var data = '';
                res.on('data', function (chunk) {
                    data += chunk;
                });

                res.on('end', function () {
                    cb(null, data);
                });
            })
                .on('error', function (err) {
                    cb(err);
                })
                .end();
        }
        else {
            cb(new Error('Failed'));
        }
    }
};

And use it like this:

app.get('/:directory', function (req, res) {
    scraper.extractEmail(req.params.directory, function(err, n) {
        if (err) return console.error(err);
        console.log(n);
        res.send(n);
    });
});

NodeJS http.request not returning data even after specifying return on the 'end' event

3 Answers3

Linked