4

I'm "walking" a hundred thousand JSON files, reading the content and throwing an error if something bad happens:

walk(__dirname + '/lastfm_test', 'json', function (err, files) {
    files.forEach(function (filePath) {
        fs.readFile(filePath, function (err, data) {
            if (err) throw err;
        });
    });
});

The walk function is largely inspired by this question (chjj answer). After some iteration, the line if (err) throw err gets executed. the error throw is:

Error: OK, open 'path/to/somejsonfile.json'

Any chance to investigate what's happening here? I'm sure that the walk function is ok: in fact replacing the call fs.readFile() with console.log(filePath) shows the paths. without errors.

Some useful info: Windows 7 x64, node.exe x64 .0.10.5. Last.fm dataset downloaded from here.

Community
  • 1
  • 1
gremo
  • 47,186
  • 75
  • 257
  • 421
  • I think you have a similar problem: http://comments.gmane.org/gmane.comp.lang.javascript.nodejs/38045 Probably you are reaching a limit of simultaneously open files on windows. – Fox32 May 07 '13 at 18:00
  • @Fox32 thank you for pointing me in the right direction. Do you know can I limit this number? – gremo May 07 '13 at 18:02
  • Both have the same problem, but no solutions... http://stackoverflow.com/questions/14694969/fs-readfile-anomaly & http://stackoverflow.com/questions/10210978/createreadstream-error-ok-open – Fox32 May 07 '13 at 18:06

2 Answers2

5

I recommend using the graceful-fs module for this purpose. It will automatically limit the number of open file descriptors. It's written by Isaac Schlueter, the creator of npm and maintainer of Node, so it's pretty solid. The bare fs module lets you shoot yourself in the foot.

Myrne Stol
  • 11,222
  • 4
  • 40
  • 48
3

The "foreach-loop" is executing readFile very often. NodeJS starts opening the files in a background thread. But no file is processed in the NodeJS main thread until the foreach loop is finished (and all file open requests are scheduled). For this reason no files are processed (and later closed) while opening all files. At some time point many files are opened and all available handles are used, resulting in the useless error message.

Their are multiple soulutions to your problem:

First you could open all files synchronously after each other. But this would slow down the application and would not match the event based programming model of NodeJS. (But is the easiest solution if you don't mind the performance)

Better would be opening only a specific amount of files at a time (e.g. ~1000 files) and after processing one you could open the next one.

Pseude Code:

1. walk the file system and store all file name in an array
2. fs.readFile for a batch of files from the array
3. In the callback of readFile after processing, start opening more files from the array if not empty.
Fox32
  • 13,126
  • 9
  • 50
  • 71