Creating a Node.js Readable stream from a JavaScript object — The simplest possible way

Question

Can anyone explain why I'm getting an error after running the following code with node v0.10.21?

I'm aware that I could JSON.stringify() that object in order to achieve more or less the same result, the point here is to make sense of stream.Readable when objectMode is set true.

The error:

net.js:612
    throw new TypeError('invalid data');
          ^
TypeError: invalid data
    at WriteStream.Socket.write (net.js:612:11)
    at write (_stream_readable.js:583:24)
    at flow (_stream_readable.js:592:7)
    at _stream_readable.js:560:7
    at process._tickCallback (node.js:415:13)
    at Function.Module.runMain (module.js:499:11)
    at startup (node.js:119:16)
    at node.js:901:3

The code:

var stream = require('stream');
var edad = require(__dirname + '/public/edad.json');

var rs = new stream.Readable({ objectMode: true });

rs.push(edad);
rs.push(null);
rs.pipe(process.stdout);

for further readers: wrote an answer in another question regarding similar context. Link: (http://stackoverflow.com/a/35886734/1453339) — Abu Shumon, Mar 09 '16 at 09:13

score 29 · Accepted Answer · answered Jan 15 '14 at 01:01

29

rs is an objectMode stream, but process.stdout is not, so it is expecting to have Buffer instances written into it. Since it is getting the wrong data type, it is throwing an error.

If you wanted to be able to pipe the objects like this, you would need to have an in-between stream that supports writing as objects and reading as Buffers.

Something like this:

var stream = require('stream');
var util = require('util');

function StringifyStream(){
    stream.Transform.call(this);

    this._readableState.objectMode = false;
    this._writableState.objectMode = true;
}
util.inherits(StringifyStream, stream.Transform);

StringifyStream.prototype._transform = function(obj, encoding, cb){
    this.push(JSON.stringify(obj));
    cb();
};


var edad = require(__dirname + '/public/edad.json');

var rs = new stream.Readable({ objectMode: true });
rs.push(edad);
rs.push(null);

rs.pipe(new StringifyStream()).pipe(process.stdout);

answered Jan 15 '14 at 01:01

loganfsmyth

156,129
30
331
251

Anyway I'm still stringifing that object right? Is there any way to just dump the object into the destination leaving with this guy the job of making it a string? – n370 Jan 16 '14 at 21:33
@n370 Not sure I follow. That's what you'd get if you just return `rs`. You just can't pipe `rs` directly to a stream expecting a string. You could certainly leave it up to the destination stream to accept an object though. – loganfsmyth Jan 16 '14 at 23:20
Oh I got it! so in our case here, can we make `process.stdout` expect an object changing it into a string before output to `stdout`? can we make it with pure Node and JS? I really appreciate your attention @loganfsmyth – n370 Jan 17 '14 at 15:36
1

Unfortunately no, that's why you need an intermediate like `StringifyStream`. You could potentially write your stream to support both, but the reality is that is really depends on your usecase. Someone could just as easily want to encode as XML instead, so it doesn't make sense to have `stdout` know anything about that. – loganfsmyth Jan 17 '14 at 16:55

slang · Answer 2 · 2015-07-21T15:37:35.450

4

As loganfsmyth noted, rs is in objectMode, while process.stdout isn't. So it is expecting to have Buffers/Strings written to it, and is throwing a TypeError when it gets an Object.

We need to convert the stream of Objects into text, which is what JSONStream is made to do:

JSONStream = require('JSONStream');
rs.pipe(JSONStream.stringify()).pipe(process.stdout);

edited Jul 21 '15 at 15:37

answered Mar 27 '15 at 23:15

slang

626
7
26

Petr Cibulka · Answer 3 · 2016-01-16T22:22:01.823

3

I am a person who does not know much about Node (I just need to have my Gulp setup working). Because of that, the fact that I need 20+ lines of code and weird calls like stream.push(null) to work with variable in the stream seems a little crazy.

This worked for me:

var data = { "prop1": "val1", "prop2": "val2" };
var dest = 'path/to/dest/config.json';
var s = require('stream');

var stream = new s.Readable();
stream.push(JSON.stringify(data));    // stream apparently does not accept objects
stream.push(null);                    // no idea why this is needed, but it is

return stream
    .pipe()                           // whatever you need
    .pipe($.fs.createWriteStream(dest));

edited Jan 16 '16 at 22:22

answered Jan 16 '16 at 22:03

Petr Cibulka

2,452
4
28
45

2

I tried editting the answer to improve it, but it seems that the queue is full. The `stream.push(null)` actually represents the end of the stream (Like `\0`). Also, Readable stream apparently is an array of strings that is converted to stream, so all elements must be strings. – Alexander Santos Jul 13 '21 at 18:08

Creating a Node.js Readable stream from a JavaScript object — The simplest possible way

3 Answers3