Is there any way to save a https response in a varable?

Question

I'm working right now with the GitHub APi, node.js in JavaScript and the https module. I need to get the total number of commits from a repository. There is no variable that says the number of it, so I'm trying to get all the commits and just count it, but I need to iterate over them, because they are paginated

Im trying to get data from the gitHub api in a function and trying to save in a local variable, but when the request ends the variable is void . I've tried to save the data in a global variable, in a variable declared before the function, trying to save it on a local file. But Im not able to get the information. Also I tried to return the data in the function, but I could not

    function getCommits(repository){
     let options {
          host: host,
           path: '/repos/'...
      }
     let request = https.request(options , (response) => {
            let body = '';
            response.on('data', (out) => {
                body += out;
                  } 
            });

     response.on('end', (out) => {
                json = JSON.parse(body);
                var i = json.length //This is the variable that I need to get out 
                                    //  from the function

            });
    }

The variable is undefined or is just the same value as I declared in the beginning

Answer 1

You have a mix of issues going on. For starters, I'd recommend you read this answer How do I return the response from an asynchronous call as it should help you to understand some of the issues with returning a value that was retrieved asynchronously (can only be returned via callback or promise).

Then, I'd suggest using the request-promise library instead of plain http.request() because it's both higher level (accumulates the entire response for you), will automatically parse to JSON and it supports a promise interface which will be really useful in looping for each page.

Then, we'll use async and await to make a loop using asynchronous operations a lot simpler.

And, lastly, you don't show exactly how a loop over all the pages would work, so I'll just give you some pseudo-code for that and you will have to fill in the real implementation for github.

const rp = require('request-promise');

async function getAllCommits(baseURL) {
    // init counter
    let totalCommits = 0;

    // page number counter
    let pageNum = 0;

    // loop until we break out
    while (true) {
        // request desired page, auto parse JSON response
        try {
            // make github request, putting page in query string
            // the exact request is pseudo-code here, you have to construct
            // the desired URL you need for your specific request
            let data = await rp({uri: baseURL, json: true, qs: {page: pageNum++}});
        } catch(e) {
            // have to decide what to do about an error here (continue, abort, etc..)
            // for now, I'll just stop further processing, assuming you've requested
            // a page beyond what exists
            // But, this needs a more real error handling strategy based on what
            // github returns
            console.log(e);
            if (real error) {
                // stops further promising, rejects async promise
                throw e;
            } else {
                // breaks out of while loop
                break;
            }
        }
        // now iterate over the data and update the totalCommits variable

    }
    return totalCommits;
}

// sample usage
getAllCommits("http://somehost/someRepository").then(total => {
    console.log(total);
}).catch(e => {
    console.log(e);
});

Some things you might not know about the getAllCommits() implmentation:

1. It's an async function which always returns a promise and it resolves that promise with whatever value you return from that function.
2. If you throw in an async function that will reject the returned promise with the exception value as the rejection reason.
3. Only async functions can use await.
4. await pauses the internal execution of the function until the promise you are awaiting resolves or rejects.
5. await only does something useful when awaiting a promise.
6. If the promise you are awaiting resolves successfully, you will get a value from the await statement.
7. If the promise you are awaiting rejects, it will essentially throw and the way to catch that error is to use try/catch with similar syntax to synchronous exceptions.
8. An async function pauses it's internal execution for await statements, but upon the first await, it immediately returns a promise to the outside world and the outside world continues to run your Javascript. So, an async function does not pause your whole sequence of execution, just the internals of the async function.
9. This means that the result of an async function is a promise and the caller must either use await on that promise or .then() on that promise to get the returned result.

Answer 2

Try writing this, the on() function will return the value of length and then we are return that value from the function.

return response.on('end', (out) => {
   json = JSON.parse(body);
   return json.length;  
});