fs.createReadStream and .pipe()

Question

I'm trying to post a large file to another server using request.post() of the request package in Node.js and I'm getting:

(node) warning: Recursive process.nextTick detected. This will break in the next version    of node. Please use setImmediate for recursive deferral.

RangeError: Maximum call stack size exceeded

I thought it's because it's a big file (80MB) because the smaller files work fine, so I am now trying to implement a stream:

var options = {
    url: aURLIHaveDefined,
    headers: {
        'User-Agent': 'MyRestAPI'
    },
    auth: {
        username: creds.username,
        password: creds.password
    },
    strictSSL: false,
    json: true
}

var readStream = fs.createReadStream(filePathIHaveDefined);

readStream.pipe(options);

And I get the error:

TypeError: Object #<Object> has no method 'on'

Does anyone know the syntax to include options in the pipe and why I'm getting this error?

UPDATE (after sousa comment): I tried:

readStream.pipe(request(options));
readStream.on('end', function() {
     console.log('end of readStream'+fileName);
});

It doesn't error out but it also doesn't seem to do anything now. The program just executes and I never see the console.log() ?

UPDATE #2:

Using the help manual here, found that you can do this:

fs.createReadStream('file.json').pipe(request.put('http://....'))

So I added the request.post() inside the pipe and it seems like I'm almost there! Using:

readStream.pipe(
    request.post(options, 
         function(error, response, data) {  

                if (error) {
                    console.log(": Cannot upload file to " + creds.server + " due to " + error);
                    setImmediate(failureCallback(error)); 
                } else {
                    console.log(": file uploaded to " + data);
                    // send back data, which is url of uploaded file
                    setImmediate( successCallback(data) ); 
                }
            }
        )
    );

I get an error saying the length needs to be known:

Apache Tomcat/5.5.17 - Error report HTTP Status 411 - Length Required his request cannot be handled without a defined content length (Length Required).

score 5 · Answer 1 · edited Apr 22 '15 at 15:03

5

The options var needs to be a stream. You have to do something like this:

readStream.pipe(request(options))

You're piping to an object, that's why you have no method on.

edited Apr 22 '15 at 15:03

Yves M.

29,855
23
108
144

answered Feb 21 '14 at 13:33

sousa

166
1
5

Your end event is not being fired. Probably your request is not working properly, meaning that the readStream never reaches the end. This is a [related post](http://stackoverflow.com/questions/15163567/events-for-fs-createreadstream-not-always-being-called) – sousa Feb 22 '14 at 14:45
try to get your `end` event from the read file stream like this: readStream.on('end', function () { console.log('end!') }); readStream.pipe(process.stdout); If this works, then you're probably doing something wrong with your `request`. – sousa Feb 22 '14 at 14:53
Hey I think the problem was that it needs to be request.post() so I actually post something but I still get the length error outlined above. Thanks for sticking with this we are close! – SOUser Feb 22 '14 at 15:01
I'm trying this form-data approach now: https://github.com/mikeal/request/issues/396 – SOUser Feb 22 '14 at 15:24

score 0 · Answer 2 · edited Apr 10 '15 at 17:47

0

For Update #2: Request github says "request can also pipe to itself. When doing so, content-type and content-length are preserved in the PUT headers"

I do not need to set headers, however if Apache Tomcat is complaining you can always set the Content-Length as follows:


    var stats = fs.statSync('file.json');

And then in the option headers:


    Content-Length':stats.size

edited Apr 10 '15 at 17:47

Brian

14,610
7
35
43

answered Apr 10 '15 at 17:26

user2821716

51
1
1

fs.createReadStream and .pipe()

2 Answers2