Node.js filter/extract subset of data from csv / JSON

Question

I am trying to extract information from a csv file, the file has many rows and I would like to return only some values/columns from a specific row. I use papa/babyparse to convert the csv file to JSON but struggle to display/extract a particular row.

var baby = require('babyparse');

var csv2 = baby.parseFiles("netreqs.csv",{
        header:true,
        skipEmptyLines: true,
        step: function(row) {
            console.log("Row:", row.data); 

        },
        complete: function() {
            console.log("All done!");
        }
});

The output I get seems to be nice JSON.

   Row: [ { Req: 'RQ0342384',
    'Requestor country': 'UK',
    ReqType: 'other',
    'ATOS Approved': '21.09.2016',
    Urgent: 'No',
    Assignee: 'Hans Gans',
    'Change number': 'NA',
    'Implementation Date': '',
    'Change fully approved': 'No',
    'Completion Date': '',
    'Req Closed': 'No' } ]
    Row: [ { Req: 'RQ0343423',
    'Requestor country': 'US',
    ReqType: 'Firewall',
    'ATOS Approved': '04.11.2016',
    Urgent: 'No',
    Assignee: 'Peter Mueller',
    'Change number': 'C9343449',
    'Implementation Date': '',
    'Change fully approved': 'No',
    'Completion Date': '31.01.2017',
    'Req Closed': 'No' } ]
...

I tried to use row.data.req for my "if" but get an "undefined" back. Also tried it with .filter and .hasOwnProperty but somehow I seem to miss something (tried also to JSON.stringify prior to the if but without success). After many hours of trail & error and googling I thought I ask here.

Idealy I'm able to use a variable to filter the "row" by Req (this is the input I get from another function) and then query other key/value pairs from this "row" as I would like to implement different responses based on the data.

I have to admit I'm fairly new to this, appreciate your support. Many thanks

Do you only need values from a single row? – cbronson Jan 03 '17 at 01:22 — cbronson, Jan 03 '17 at 01:22

Celso Agra · Accepted Answer · 2017-01-04T14:55:43.310

0

I believe you don't need use a parser for that. Just need to use the readLine:

var output = [];
var count = 0

var lineReader = require('readline').createInterface({
  input: require('fs').createReadStream('file.csv')
});

lineReader.on('line', function (line) {
  var jsonFromLine = {};

  if (count == 2) { // this is my conditional. Set line 2
    var lineSplit = line.split(',');
    // select columns you want
    jsonFromLine.column0 = lineSplit[0];
    jsonFromLine.column1 = lineSplit[1];
    // ...  
    output.push(jsonFromLine);
  }
  count++;
});

lineReader.on('close', function (line) {
    console.log(output); // list output 
});

I hope it helps.

Edit.:

If you need a specific value, you can set a different conditional:

var output = [];

var lineReader = require('readline').createInterface({
  input: require('fs').createReadStream('file.csv')
});

lineReader.on('line', function (line) {
    var jsonFromLine = {};
    var lineSplit = line.split(',');
    // select columns you want
    jsonFromLine.req = lineSplit[0];
    jsonFromLine.column1 = lineSplit[1];
    // ...  
    if (jsonFromLine.req === 'RQ0191223') {
        output.push(jsonFromLine);
    }
});

lineReader.on('close', function (line) {
    console.log(output); // list output 
});

this is works fine to me

Edit.2:

You can use promises as well:

var method = function () {
    return new Promise(function(resolve) {
        var output = [];
        var lineReader = require('readline').createInterface({
          input: require('fs').createReadStream('file.csv')
        });

        lineReader.on('line', function (line) {
            var jsonFromLine = {};
            var lineSplit = line.split(',');
            // select columns you want
            jsonFromLine.req = lineSplit[0];
            jsonFromLine.column1 = lineSplit[1];
            // ...  
            if (jsonFromLine.req === 'RQ0191223') {
                output.push(jsonFromLine);
            }
        });

        lineReader.on('close', function (line) {
            resolve(output);
        }); 
    });
}


method().then(function(outputOfResolve) { console.log(outputOfResolve); });

I hope it helps

edited Jan 04 '17 at 14:55

answered Jan 03 '17 at 01:30

Celso Agra

1,389
2
15
37

thank you for your fast reply but I get an error message: TypeError: reader.on is not a function – DPM Jan 03 '17 at 09:18
Ops. I changed the variable now. – Celso Agra Jan 03 '17 at 10:08
you can add a conditional on your code. I put a conditional related to the number of line, but you can change for another kind of conditional – Celso Agra Jan 03 '17 at 12:00
cheers. sorry if it gets a bit silly for you but how would I test for a particuar value from within the file? "if (jsonFromLine.req == 'RQ0191223')" seems to not work – DPM Jan 03 '17 at 13:17
how would I now pack that into a function that I could call from another script? – DPM Jan 04 '17 at 08:48
you can put this code in the `var methodName = function ( /* args */ ) { /* code */ }`. You can set some actions in the `lineReader.on('close', ...) `. Can you show some examples how you plan to use this method? – Celso Agra Jan 04 '17 at 10:02
Thanks Celso, please find it here http://stackoverflow.com/questions/41453982/node-js-resolve-promise-and-return-value – DPM Jan 04 '17 at 10:29
I added a new code above! But I believe you can use Promises as @Sergeon said. – Celso Agra Jan 04 '17 at 14:54
cheers @Celso Agra. That works great if I run the code from within the same script but how would I call "method()" from another file? (I renamed method to getReqStatus but always get the error message "getReqStatus is not a function") – DPM Jan 04 '17 at 19:32
You have to set `module.exports = function () {.....}`, instead of `var method = function () {.....}`, in the *reqstatus.js*. – Celso Agra Jan 04 '17 at 19:50
my bad... It works fine now. The main issue was that the variable that my main script had to pass on was not ok. Thanks a lot for your supprot @Celso Agra! – DPM Jan 04 '17 at 23:44

Node.js filter/extract subset of data from csv / JSON

1 Answers1