Recursive Ajax deferred objects

Question

I have the following situation. I would like to load file A from a server which in turn will try to load files A1, A2, A3, ... An and each file A[1-n] will in turn load other files and this can keep going; but there is an end to it. I would like to set it up using deferred objects (so as to not hang the browser by using async: false) but the recursion of loading and parsing the files confuses me on how to set up the objects. In addition there is a requirement that the highest recursion depth level (l) must finish before I can continue with level (l-1). For the case where there is no recursion this code works but the recursive case eludes me.

var loadFile = function (index, url, scope, callback) {
    $.ajax ({url: url, type: 'GET', dataType: 'text'})
    .done (function (responseText) {
        // store response in array
        scope.requests[index] = responseText;
    })
    .fail (function (xhr, status, error) {
        scope.requests[index] = 'error';
    })
    .always (function (responseText) {
        // loop through entire response array from the beginning
        for (var j = 0; j < scope.requests.length; j++)
        {
            if (scope.requests[j] === 'unprocessed') return;
            else if (scope.requests[j] === 'error')
                scope.requests[j] = 'done';
            else if (scope.requests[j] !== 'done')
            {
                parseFile (scope.requests[j], scope, callback);
                scope.requests[j] = 'done';
            }
        }

        // if all are done then reset the array and run the callback
        delete scope.requests;

        if (callback) callback();
    });
}

var parseFile = function (responseText, scope, callback) {
    var lines = responseText.split("\n");

    for (var l = 0; l < lines.length; l++) {
        var line = lines[l];
        line = line.replace (/^\s+/, ''); // remove leading white space
        if (line.charAt(0) === '1') // file reference
        {
            var attrs = line.split (/\s+/);

            // the file will exist in any of the paths in pathList
            for (var i = 0; i < scope.pathList.length; i++) {
                scope.requests.push ('unprocessed');
                loadFile (++index, scope.pathList[i] + attrs[14], scope, callback);
            }
        }
    }
}

var index = 0;
var this.requests = [];
this.requests.push ('unprocessed');
loadFile (index, fileAi, this, callback);

Answer 1

The basic idea is this:

1. Send ajax request for each file in the current level.
2. When entire level is done, parse all the responses.
3. If there are more, recurse. If not, call callback.

Since some of the files you are requesting do not exist (and this is expected), you need to create your own Deferreds. Then you can resolve them from the ajax done and fail callbacks, effectively ignoring the failures.

Also, I added a cache object as you requested. The object maps urls to promises. When you attach a done callback to a promise that is already resolved, the callback is called right away with the same response argument. This is a nice way to cache things, since the first request doesn't have to be finished for it to be in the cache, since you are caching the request instead of the response. So if you request the same file 4 times before the first request even finishes, it will still only result in one ajax call.

Note: Since I added the function getFile, the scope/closure issue from our comments are no longer a problem (since each dfd variable is now in a function scope), so the code is a bit less confusing, I think. The problem was the very common loop scope issue.

Code:

// load all files, starting with startUrl.  
// call callback when done.
var loadAll = function(startUrl, callback) {
    var pathList = []; // assuming this has some base urls in it.
    var dfds = []; // dfds for the current level.
    var urls = [startUrl]; // urls for current level.
    var responses = []; // responses for current level.
    var cache = {}; // object to map urls to promises.

    // given the responseText, add any referenced urls to the urls array
    var parseFile = function (responseText) {
        var lines = responseText.split("\n");

        for (var l = 0; l < lines.length; l++) {
            var line = lines[l];
            line = line.replace (/^\s+/, ''); // remove leading white space
            if (line.charAt(0) === '1') // file reference
            {
                var attrs = line.split (/\s+/);

                // the file will exist in any of the paths in pathList
                for (var i = 0; i < pathList.length; i++) {
                    // add new path to urls array
                    urls.push (pathList[i] + attrs[14]);
                }
            }
        }
    };

    // load one file.
    // check cache for existing promise for the url.
    var getFile = function(url) {
        var dfd;

        if(cache.hasOwnProperty(url)){
            // use cached promise.
            // if it is already resolved, any callback attached will be called immediately.
            dfd = cache[url];
            dfds.push(cache[url]);
        } else {
            dfd = $.Deferred();
            $.ajax ({url: url, type: 'GET', dataType: 'text'}).done(function(response){
                // resolve and pass response.
                dfd.resolve(response);
            }).fail(function(){
                // resolve and pass null, so this error is ignored.
                dfd.resolve(null);
            });
            dfds.push(dfd.promise());
            cache[url] = dfd.promise();
        }

        // when the request is done, add response to array.
        dfd.done(function(response) {
            if(response){
                // add to responses array.
                // might want to check if the same response is already in the array.
                responses.push(response);
            }
        });
    };

    // request each file in the urls array.
    // recurse when all requests done, or call callback.
    var loadLevel = function () {
        dfds = [];
        responses = [];

        for (var l = 0; l < urls.length; l++) {
            getFile(urls[l]);
        }

        $.when.apply($, dfds).done(function(){
            // entire level is done loading now.
            // each done function above has been called already, 
            // so responses array is full.
            urls = [];

            // parse all the responses for this level.
            // this will refill urls array.
            for (var i = 0; i < responses.length; i++) {
                parseFile(responses[i]);
            }

            if(urls.length === 0) {
                // done
                callback();
            } else {
                // load next level
                loadLevel();
            }
        });
    };

    // load first level
    loadLevel();
};

Answer 2

I don't think you'll be able to achieve a "level by level" block with the code structured as is, because as written the code will always try to complete an entire branch before the recursion unwinds, i.e. given this structure:

     1
    / \
   2   6
  /|\  |
 3 4 5 7

it would follow the nodes in the numeric order shown, not [1] [2 6] [3 4 5 7] (or did you perhaps mean [3 4 5 7] [2 6] [1]? )

I can't offer a complete solution, just a few hints that I think will help.

1. you'll need to create an array for each level, contained a deferred object for each requested file at that level.

2. you can't use the jqXHR object for that because you also recurse in the .fail case, so you'll have to create a separate $.Deferred() yourself and then .resolve that inside your .always handler.

3. use $.when.apply($, myArray).done(...) to trigger callbacks that only happen when all of the elements in myArray have been completed.