async.waterfall duplicates calls

Question

I have the below async code.

  for fileName in sourceFiles
    console.log 'dealing with ', fileName
    async.waterfall [
      (callback) ->
        console.log "going to read ", fileName
        fs.readFile fileName, (err, content) ->
          if err then throw err
          callback null, fileName, content
          return
      (fileName, content, callback) ->
        console.log 'length of: ',  fileName, ' is: ', content.length

I expected that the output will be something like:

dealing with file1
going to read file1
length of file1 is 10
dealing with file2
going to read file2
length of file 2 is 20

instead, what I get is this:

dealing with file1
dealing with file2
going to read file2
going to read file2 <- note it is the same file repeated
length of file2 is 20
length of file2 is 20

I'm not able to figure out why this would be so. (This being a coffeescript is no issue. It is the same output in JS too)

score 1 · Answer 1 · answered Jul 23 '14 at 12:40

The async library doesn't duplicate the calls, but you have an issue with scoping here.

The fileName in your loop already is set to file2 when your first function is called you should try to wrap that into another function, just to get a new scope to work with, like this:

for fileName in sourceFiles
  (fileName) ->
    console.log 'dealing with ', fileName
    async.waterfall [
      (callback) ->
        console.log "going to read ", fileName
        fs.readFile fileName, (err, content) ->
          if err then throw err
          callback null, fileName, content
          return
      (fileName, content, callback) ->
        console.log 'length of: ',  fileName, ' is: ', content.length

score 0 · Answer 2 · answered Jul 23 '14 at 13:03

It appears that async.waterfall is asynchronous too so your loop is terminated before the waterfall callback and waterfall is called 2 times with the same fileName.

But here, waterfall is useless, you can simply use readFileSync who is synchronous.

http://nodejs.org/api/fs.html#fs_fs_readfilesync_filename_options

async.waterfall duplicates calls

2 Answers2