NodeJS streaming gzip to a file

Question

I am trying to write gzip'd content to a file to make the file smaller.

I seem to be missing a step - when I read back the file, the text is plain text (e.g. hello there), instead of gzip'd content (e.g. \u001f\ufffd\b\u0000\u0000\u0000...).

I've written the following unit test to reproduce what I'm doing - hopefully someone can point out where I went wrong.

NodeJS: v18.3.0

import fs from 'fs'
import path from 'path'
import zlib from 'zlib'

// ...

it('wip general - check compression to file works', async () => {
    // create gzip pipe to file
    const filename = path.join(tmpdir.name, 'test.txt.gz')
    const test = zlib.createGzip().pipe(fs.createWriteStream(filename, 'utf-8'))

    // write some content, wait for it to finish writing
    test.write('hello there', 'utf-8')
    test.end()
    await new Promise(resolve => test.on('finish', resolve))

    // check result is NOT plaintext
    expect(fs.readFileSync(filename, 'utf-8')).not.eql('hello there')
})

Failure output:

AssertionError: expected 'hello there' to not deeply equal 'hello there'

Nick Grealy · Answer 1 · 2022-08-26T01:55:39.727

Ok, worked it out.

When you call the .pipe() function, it returns the parameter you pass in... which means:

const test = zlib.createGzip().pipe(fs.createWriteStream(filename))

is actually setting test to the createWriteStream, completely bypassing the gzip stream.

The solution was to store the separate stream objects, to ensure we're writing to the gzip stream.

i.e.

it('wip general - check compression to file works', async () => {
    const filename = path.join(tmpdir.name, 'test.txt')
    const out = fs.createWriteStream(filename, 'utf-8')
    const gzip = zlib.createGzip()
    gzip.pipe(out)
    gzip.write('hello there', 'utf-8')
    gzip.end()
    await new Promise(resolve => out.on('finish', resolve))

    // check result is encoded
    expect(fs.readFileSync(filename, 'utf-8')).not.eql('hello there')
})

NodeJS streaming gzip to a file

1 Answers1