How to use await on https module to get response in node.js

Question

why does the 1st log print response data, and 2nd log prints function definition instead of response data. is there a way to debug and see the flow?

const https = require("https");
const substr = "spider";
const pageNum = 1;

let list = [];

const fetch = (url) =>
  https.get(url, async (res) => {
    res.setEncoding("utf-8");
    let ttt;
    await res.on("data", function (data) {
      ttt = data;
    });
    console.log(ttt); //1st log
    return ttt;
    
  });

  
((substr) => {
  totalPages = 1;

  pageNum;
  for (let i = 1; i <= totalPages; i++) {
    const res = fetch(
      "https://jsonmock.hackerrank.com/api/movies/search/?Title=" +
        substr +
        "&page=" +
        i
    );
    console.log(res);//2nd log
  }
})("s");

Does this answer your question? [How do I return the response from an asynchronous call?](https://stackoverflow.com/questions/14220321/how-do-i-return-the-response-from-an-asynchronous-call) — jonrsharpe, Jan 14 '21 at 20:20

Michael Zigldrum · Answer 1 · 2021-01-15T06:48:42.487

There are some more reasons besides the missing await statement before the call to fetch. The reason is because the fetch function does not in fact return a promise.

You set it as an arrow function. Arrow functions will do an implicit return if the function body only has a single expression. So the fetch function returns the result of the https.get() function call, which is just the function signature as it works with the callback concept.

Additionally you did not handle the event listener correctly, as the res.on('data', <function here>) line does not return a promise, so the await keyword did not do anything.

Let's see how I would rewrite your code to run as expected:

const https = require('https')
const substr = 'spider'
const pageNum = 1

let list = []

// Create an async function fetch that takes the url
const fetch = (url) => {
  // Return a promise
  return new Promise((resolve, reject) => {
    // Call the https.get function
    https.get(url, (res) => {
      res.setEncoding('utf-8')
      // Set the event listener
      res.on('data', function (data) {
        // Here the actual data event is _happening_. Now we have the return value
        console.log(data) // 1st Log
        resolve(data)
      })
    })
  })
}

(async (substr) => {
  totalPages = 1;

  pageNum;
  for (let i = 1; i <= totalPages; i++) {
    // await the promise returned from fetch
    const res = await fetch(
      "https://jsonmock.hackerrank.com/api/movies/search/?Title=" +
      substr +
      "&page=" +
      i
    );
    console.log(res);//2nd log
  }
})("s");

I hope this attempt at an explanation helps!

Just a tiny nit, that ``async`` keyword at the start of the ``fetch`` function is not necessary. — Take-Some-Bytes, Jan 15 '21 at 01:04
Yes true, I just got used to add async to all methods. I'll edit my answer — Michael Zigldrum, Jan 15 '21 at 06:48

score 1 · Answer 2 · answered Jan 16 '21 at 15:08

found it. looks like https doesn't return anything, you can use Promise/resolve to return response after the request is completed.

const https = require("https");
const substr = "spider";
const pageNum = 1;

let list = [];


const fetch = async (url) => {
  return new Promise((resolve) => {
    https.get(url, async (res) => {
      res.setEncoding("utf-8");
      let ttt;
      await res.on("data", function (data) {
        ttt = data;
        resolve(data);
      });
      res = ttt;
      console.log(ttt);
    });
  });
};

(async (substr) => {
  totalPages = 1;

  pageNum;
  for (let i = 1; i <= totalPages; i++) {
    const res = await fetch(
      "https://jsonmock.hackerrank.com/api/movies/search/?Title=" +
        substr +
        "&page=" +
        i
    );
    console.log(res);
  }
})("s");

score 0 · Answer 3 · answered Jan 14 '21 at 20:24

0

await is a keyword that is used in combination with a Promise value. It tells the code to wait until the Promise resolves a value, or tells it to throw if there is an error. If you check the function definition for fetch you will see that it returns a Promise instead of the response object directly. You need to use the await keyword to resolve that Promise.

const res = await fetch(...

answered Jan 14 '21 at 20:24

Deadron

5,135
1
16
27

I think the problem is also in the fetch function. Just adding `await` before the call to fetch will not solve the problem. the fetch function is not returning a promise, as https.get() is using the callback concept. I made an alternative solution that actually returns a new promise in fetch. Also I find the event listener handling would not work as written in the question – Michael Zigldrum Jan 14 '21 at 21:01
ah I read fetch as being one of the numerous nodejs http client libraries. Most of them return promises. I see now you are using https directly – Deadron Jan 14 '21 at 21:02

score 0 · Answer 4 · answered Jan 14 '21 at 20:24

0

You are not awaiting the fetch request in the loop. Res just holds the Promise that fetch returns. You instead need to have a try and catch block around the statement with an await before the fetch call or you simply use .then() and .catch().

answered Jan 14 '21 at 20:24

FireFighter

686
3
6

How to use await on https module to get response in node.js

4 Answers4