Async Recursive Readdir in Node

Question

I'm trying to scan a large directory of files and sub-directories with a recursive function that I've written. It's very similar to the parallel loop given in this answer.

var fs = require('fs');
var walk = function(dir, done) {
  var results = [];
  fs.readdir(dir, function(err, list) {
    if (err) return done(err);
    var pending = list.length;
    if (!pending) return done(null, results);
    list.forEach(function(file) {
      file = dir + '/' + file;
      fs.stat(file, function(err, stat) {
        if (stat && stat.isDirectory()) {
          walk(file, function(err, res) {
            results = results.concat(res);
            if (!--pending) done(null, results);
          });
        } else {
          results.push(file);
          if (!--pending) done(null, results);
        }
      });
    });
  });
};

This issue is that it isn't really asynchronous. The whole thing processes and returns a giant array of files. Is there a way to recursively scan a directory asynchronously?

Something more like:

walk(path,function(files) {
  // do stuff with each file as its found
});

EDIT: once I start getting the files, I plan to access them and use the async module to process them and prevent using up file descriptors. So something like this:

walk(path,function(files) {
    async.each(files,function() {
        // do more stuff
    }
});

Will that work okay with an asynchronous scan?

Your code is already asynchronous. I think what your'e after is creating your code in a module such that you can extend the event emitter object and then register to be notified when each file has been read. http://code.tutsplus.com/tutorials/using-nodes-event-module--net-35941 — HeadCode, Dec 09 '14 at 06:36

score 1 · Accepted Answer · answered Dec 09 '14 at 07:01

yes HeadCode already explained it in a comment above. You can use eventEmitter to do this kind of Asynchronous recursive stuff. For your code you can put the walk function as an event callback.

var EventEmitter = require("events").EventEmitter;

var ee = new EventEmitter();
ee.on("walk", YourWalkFunction);

ee.emit("walk",YourParams);

Async Recursive Readdir in Node

1 Answers1