3

I would have expected that I got all the CSV data in the array, but for some reason I don't get anything.

Can someone explain why a is empty at the end?

const fs = require('fs');
const { parse } = require('csv-parse');

let a = [];

fs.createReadStream('./example.csv')
   .pipe(parse({ delimiter: ';', from_line: 2 }))
   .on('data', function (row) {
      a.push(row);
   })
   .on('end', function () {
      console.log('finished');
   })
   .on('error', function (error) {
      console.log(error.message);
   });

console.log(a);
Sandra Schlichting
  • 25,050
  • 33
  • 110
  • 162
  • This is because `a` is logged **before** the stream is read. Move the log inside the `end` callback. – briosheje Jul 20 '22 at 11:44
  • If I replace `.push(row);` with `console.log(row)` then I see the data. Can you show what you mean by moving the log to `end`? – Sandra Schlichting Jul 20 '22 at 11:49
  • 1
    Just replace `console.log('finished')` with `console.log(a)`. Because streams are async, in your current code `console.log(a)` is executed either before the stream is read or while the stream is read. the `end` stream event should be raised once all the data has been read and the stream has been closed, hence, at such a point, `a` will be filled. If you don't feel comfortable with the callback approach, you might want to take a look at stream/promise, check this question for further informations: https://stackoverflow.com/a/65938887/2275797 . – briosheje Jul 20 '22 at 11:51

3 Answers3

2

createReadStream() runs parallel. So after calling this function, the program continues and the next process is your console.log(a). Thats the idea behind streams because the program can continue running while the big file is read out.

The code inside the .on() functions is called later. Thats why they are called "callbacks".

So you are logging the result before any data is read...

tom203
  • 77
  • 5
  • What I don't understand is that I can see the data if I replace `a.push(row)` with `console.log(row)`. – Sandra Schlichting Jul 20 '22 at 11:55
  • 1
    @SandraSchlichting okay. As I said the .on() functions are called by the createReadStream(). So the file needs to be opened etc. That takes a while. If the data is available your function inside .on('data'... is called. Thats one point. If you look at your log, you should see that console.log(a); is executed before console.log(row); That is because createReadStream() is async. It creates a new thread. So the program runs parallel. After calling createReadStream() it directly continues processing. So console.log(a); is executed. After opening the file etc. data is available and gets logged. – tom203 Jul 20 '22 at 12:00
2

As explained before the headstream runs in the background. You can wrap the stream in a promise to get the result. Here is one possible solution:

const fs = require('fs');
const { parse } = require('csv-parse');

const readFile = async () => new Promise((resolve, reject) => {
    let a = [];
    fs.createReadStream('./example.csv')
        .pipe(parse({ delimiter: ';', from_line: 2 }))
        .on('data', function (row) {
            a.push(row);
        })
        .on('end', function () {
            resolve(a);
        })
        .on('error', function (error) {
            reject(error)
        });
});

readFile().then(data => console.log(data))
nilskch
  • 307
  • 3
  • 10
2

Your a is undefined because the code is executed asynchronously. The on() function is called after your console.log(a) is executed.

You can read the value of a inside the data event or the end event:

fs.createReadStream('./example.csv')
   .pipe(parse({ delimiter: ';', from_line: 2 }))
   .on('data', function (row) {
      a.push(row);
      console.log(a) // <------- here
   })
   .on('end', function () {
      console.log('finished');
      console.log(a) // <------- here
   })
   .on('error', function (error) {
      console.log(error.message);
   });

One solution may be to use a Promise

const readCSVData = async () => new Promise((resolve, reject) => {
let a = []
fs.createReadStream('./example.csv')
   .pipe(parse({ delimiter: ';', from_line: 2 }))
   .on('data', function (row) {
      a.push(row);
   })
   .on('end', function () {
      // return the a value 
      resolve(a) 
   })
   .on('error', function (error) {
      reject(error)
   });
})

// ..... 

const foo = async () => {
  const data = await readCSVData()
  console.log(data)
}

// or
readCSVData().then(a => console.log(a))
Fabio
  • 593
  • 3
  • 14