0

I have a strange situation regarding http server and piping request.

From my past experience, when piping the request object of a http server to a writable stream of some sort, it does not include the headers, just the payload.

Today however, I wrote some very simple code, and from some reason, I'm spending the past 2 hours trying to figure out why it writes the headers to the file (super confusing!)

Here's my code:

server = http.createServer((req, res) => {
  f = '/tmp/dest'
  console.log(`writing to ${f}`)
  s = fs.createWriteStream(f)
  req.pipe(s)
  req.on('end', () => {
    res.end("done")
  })
})

server.listen(port)

I test this with the following curl command:

curl -XPOST  -F 'data=@test.txt' localhost:8080

And this is what I'm getting when I'm reading /tmp/dest:

--------------------------993d19e02b7578ff
Content-Disposition: form-data; name="data"; filename="test.txt"
Content-Type: text/plain

hello - this is some text


--------------------------993d19e02b7578ff--

Why am I seeing the headers here? I expected it to only write the payload

I have a code I wrote about a year ago that streams directly to a file without the headers, I don't understand what's different, but this one did the trick:

imageRouter.post('/upload', async(req, res) => {
  if(!req.is("image/*")) {
    let errorMessage = `the /upload destination was hit, but content-type is ${req.get("Content-Type")}`;
    console.log(errorMessage);
    res.status(415).send(errorMessage);
    return;
  }

  let imageType = req.get("Content-Type").split('/')[1];
  let [ err, writeStream ] = await getWritableStream({ suffix: imageType });
  if (err) {
    console.log("error while trying to write", err);
    return res.status(500).end();
  }

  let imageName = writeStream.getID();
  req.on('end', () => {
    req.unpipe();
    writeStream.close();
    res.json({
      imageRelativeLink: `/images/${imageName}`,
      imageFullLink: `${self_hostname}/images/${imageName}`
    });
  });
  req.pipe(writeStream);
});

What's different? Why does my code from a year ago (last block) writes without the form-data/headers? The resulting file is only an image, without text, but this time (the first block) shows http headers in the resulting file

Tom Klino
  • 2,358
  • 5
  • 35
  • 60
  • What you are seeing is the plain contents of the http protocol format. This is what it is supposed to be, this a message as per [http specification](https://datatracker.ietf.org/doc/html/rfc2616#section-4). – Edwin Dalorzo Oct 09 '21 at 16:24
  • How can I pipe just the payload then? – Tom Klino Oct 09 '21 at 16:25
  • Also, this answer says the same, only the payload should be piped: https://stackoverflow.com/a/29199616/1463751 (unless there's something I'm missing) – Tom Klino Oct 09 '21 at 16:27
  • You can’t do it directly without modifying the source stream. Alternatively you could listen to request events (e.g. `on(‘request’, fb)`) and extract just the body there and write it to your output stream. – Edwin Dalorzo Oct 09 '21 at 16:30
  • I already did it directly in a past code I wrote - I just don't understand what's different now. I'll edit my question to include the code I wrote in the past that streamed without headers – Tom Klino Oct 09 '21 at 16:33
  • Why do you need to use `pipe` when you could just as easily write the `request.body` to your output stream? You also need to handle `on(’end’, fn)` to determine when to close the output stream. – Edwin Dalorzo Oct 09 '21 at 16:38
  • Because the service I'm writing will upload large files (videos) - the text file is just a debug test – Tom Klino Oct 09 '21 at 16:40
  • And how does `pipe` help you deal with large files in a way that `on(’request, fn)` and `on(’end’, fn)` can't? – Edwin Dalorzo Oct 09 '21 at 16:45
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/237989/discussion-between-tom-klino-and-edwin-dalorzo). – Tom Klino Oct 09 '21 at 16:47

2 Answers2

1

Instead of using pipe, try using on('data') and referring to req.data to pull off the contents. This will allow the http library to process the HTTP body format and handle the "headers" (really: form part descriptors) for you.

Node Streaming Consumer API

    server = http.createServer((req, res) => {
      f = '/tmp/dest'
      console.log(`writing to ${f}`)
      s = fs.createWriteStream(f)
      req.on('data', chunk) => {
          s.write(chunk);
      }
      req.on('end', () => {
        s.close();
        res.end("done")
      })

})

server.listen(port)
Joe Herman
  • 21
  • 4
  • Thanks, but it still writes the form part descriptors... – Tom Klino Oct 09 '21 at 16:56
  • This also doesn't have any flow control (when `s.write()` returns that the stream is full). – jfriend00 Oct 09 '21 at 17:48
  • If using express http, was express told to expect encoded data? Something like: [app.use](https://stackoverflow.com/questions/4295782/how-to-process-post-data-in-node-js#:~:text=//%20Parse%20URL-encoded,body.user.email) – Joe Herman Oct 10 '21 at 03:15
0

As it turns out, I had a mistake in my understanding, and therefore made a mistake in my question.

What I thought were the headers, were actually http multipart specification. This is how curl uploads a file when used with this syntax.

What I actually needed was to change the way I test my code with curl to one of the following:

cat /path/to/test/file | curl -T - localhost:8080
# or
curl -T - localhost:8080 < /path/to/test/file
# or
curl -T /path-/to/test/file localhost:8080 < /path/to/test/file

Using the -T (or --upload-file) flag, curl uploads the file (or stdin) without wrapping it in an http form.

Tom Klino
  • 2,358
  • 5
  • 35
  • 60