How to avoid memory leak when creating promises recursively?

Question

I have an ionic application which needs to download data (with pagination) and insert it into database recursively (like a pipeline). I'm creating promises with (Angular's) $q service. The problem is that when I call this recursive function, the data is being downloaded and inserted successfully, however memory usage is always increasing, and when the promise chain resolves completely, allocated memory still remains in use.

Here is my recursive function:

// offset:  last downloaded row count
// limit:   row count to download at each page
// numRows: row count to download at all
function dowloadAndInsert(offset, limit, numRows) {
    var deferred = $q.defer();

    // Recursion step: We do not reached at the end of data
    if((offset + limit) <= numRows) {

        // Download the data
        downloadData(offset, limit)
            .then(function(response) {

                // Insert the data
                insertData(response)
                    .then(function(insertTime) {

                        // Recursion step
                        dowloadAndInsert(offset + limit, limit, numRows)
                            .then(function() {
                                deferred.resolve();
                            })
                            .catch(function(reason) {
                                deferred.reject(reason);
                            });
                    })
                    .catch(function(reason) {
                        deferred.reject(reason);
                    });
            })
            .catch(function(reason) {
                deferred.reject(reason);
            });
    }

    // Base case: We reached at the end of data
    else {
        var remainingRows = numRows % limit;        // Means the last limit actually

        // If exists, insert remaining rows
        if(remainingRows !== 0) {

            // Download the last piece of data
            downloadData(offset, remainingRows)
                .then(function(response) {

                    // Insert the last piece of data
                    insertData(response)
                        .then(function(insertTime) {

                            // Base case, successfully downloaded and inserted everything
                            deferred.resolve();
                        })
                        .catch(function(reason) {
                            deferred.reject(reason);
                        });
                })
                .catch(function(reason) {
                    deferred.reject(reason);
                });
        }

        else {
            // Base case, successfully downloaded and inserted everything
            deferred.resolve();
        }
    }

    return deferred.promise;
}

Note: response object coming from downloadData function is a big data, it sometimes contains 100.000 rows with 18 columns. Total memory usage is becoming like 1GB. I'm running my tests on iPad Air 2.

I'm playing with big data in my recursion function, so that my question is a bit different from other recursive memory leak questions.

Thanks.

Possible duplicate of [How do I stop memory leaks with recursive javascript promises?](http://stackoverflow.com/questions/15027192/how-do-i-stop-memory-leaks-with-recursive-javascript-promises) — Heretic Monkey, Aug 18 '16 at 20:13
Please do search before asking a question. The duplicate has almost exactly the same wording as yours for the title, and when I search for your title, it has the duplicate as the second hit (after this question of course). — Heretic Monkey, Aug 18 '16 at 20:14
**Avoid the [deferred antipattern](http://stackoverflow.com/q/23803743/1048572)!** The promise implementation [should take care of this itself](http://stackoverflow.com/q/29925948/1048572). — Bergi, Aug 18 '16 at 20:54
This sounds more likely to be a memory usage issue, not a memory leak issue. You should only suspect a leak if you run this operation multiple times and the total memory used by your process continues to rise every incremental time you run it. Promises take care of themselves just like any other object in Javascript does. Your job is to make sure that there are no lasting references to data you no longer need. If you do that, there will be no leaks. — jfriend00, Aug 19 '16 at 00:31
Thanks. You are right, this is not memory leak but it is memory usage issue. Finally, I solved my problem by taking download and insert operation into a separate function, instead of using big data in my recursive function. — nskarakoc, Aug 21 '16 at 20:16

Benjamin Gruenbaum · Answer 1 · 2016-08-18T22:18:59.957

3

Your code is working way too hard, promises chain, so when you do the little deferred dance - you could really just chain promises which should resolve the leak as a side effect of the better code:

function dowloadAndInsert(offset, limit, numRows) {
  const start = offset,
        numFetch = ((offset + limit) <= numRows ? limit : numRows % limit;
  if(numFetch === 0) {
     return Promise.resolve(); // we're done;
  }
  return downloadData(start, end).
           then(insertData).
           then(downloadAndInsert.bind(null, offset + numFetch, limit, numRows);
}

And that's the entire code, it says:

Check how many rows I need to fetch and insert.
If I don't need to fetch anymore rows - just return an empty promise.
Otherwise fetch as many as we need - and then fetch the remaining rows.

edited Aug 18 '16 at 22:18

answered Aug 18 '16 at 20:34

Benjamin Gruenbaum

270,886
87
504
504

This suggests a much cleaner way to write the promise code, but I'm not sure it actually addresses the real point of the question which is how much memory the operation is consuming (see last paragraph of the question) which the OP seems to think is a leak (but that is not yet shown whether this is just a memory usage issue or an actual leak). – jfriend00 Aug 19 '16 at 00:32
@jfriend00 memory usage should be much much lower when not using a 4 level nested closure which would actually be really expensive when doing recursion. Here - no closures are kept - the only tracking you need is the next promise and that only starts when this promise resolves so no memory has to be kept. – Benjamin Gruenbaum Aug 19 '16 at 00:43
Now to be perfectly fair - leak is tricky terminology - if we consider leaks only things that don't eventually get cleared up - you are right - but if we consider leaks things that consume more and more memory as an operation goes on when they don't have to - this code doesn't leak in a smart promise library which can skip intermediate promises after they resolve when tracking an end promise. I'm not sure if $q does this since I haven't looked at it in over a year - but I hope so. – Benjamin Gruenbaum Aug 19 '16 at 00:43
If you think your code recommendation helps with the memory leak/usage issue the OP has, then please add that info to the text of your answer and explain how you think the problem might be solved. I did not get that info from your answer. – jfriend00 Aug 19 '16 at 00:46
This does not solved my problem. My problem is not because of recursive promises chain, but it is because of big data usage. I've reduced memory usage by taking download and insert operations into a separate function and calling it, instead of keeping big data in my recursive function. – nskarakoc Aug 21 '16 at 20:14

How to avoid memory leak when creating promises recursively?

1 Answers1