Working with Node.js streams without callbacks

Question

To send a PDF file from a Node.js server to a client I use the following code:

const pdf = printer.createPdfKitDocument(docDefinition);

const chunks = [];

pdf.on("data", (chunk) => {
    chunks.push(chunk);
});

pdf.on("end", () => {
    const pdfBuffered = `data:application/pdf;base64, ${Buffer.concat(chunks).toString("base64")}`;
    res.setHeader("Content-Type", "application/pdf");
    res.setHeader("Content-Length", pdfBuffered.length);
    res.send(pdfBuffered);
});

pdf.end();

Everything is working correctly, the only issue is that the stream here is using callback-approach rather then async/await.

I've found a possible solution:

const { pipeline } = require("stream/promises");

async function run() {
    await pipeline(
        fs.createReadStream('archive.tar'),
        zlib.createGzip(),
        fs.createWriteStream('archive.tar.gz')
    );

    console.log('Pipeline succeeded.');
}

run().catch(console.error);

But I can't figure out how to adopt the initial code to the one with stream/promises.

score 0 · Answer 1 · edited Jun 30 '21 at 22:01

You can only convert a callback-API to async/await if the callback is intended to only be executed once.

The one you found online works, because you're just waiting for the whole stream to finish before the callback runs once. What you've got is callbacks that execute multiple times, on every incoming chunk of data.

There are other resources you can look at to make streams nicer to consume, like RXJS, or this upcoming ECMAScript proposal to add observables to the language. Both of these are designed to handle the scenario when a callback can execute multiple times — something that async/await can not do.

Thanks for the references, I'll check it! – Mike Jun 30 '21 at 22:02 — Mike, Jun 30 '21 at 22:02

jfriend00 · Accepted Answer · 2021-06-30T23:52:19.227

You can manually wrap your PDF code in a promise like this and then use it as a function that returns a promise:

function sendPDF(docDefinition) {
    return new Promise((resolve, reject) => {
        const pdf = printer.createPdfKitDocument(docDefinition);

        const chunks = [];

        pdf.on("data", (chunk) => {
            chunks.push(chunk);
        });

        pdf.on("end", () => {
            const pdfBuffered =
                `data:application/pdf;base64, ${Buffer.concat(chunks).toString("base64")}`;
            resolve(pdfBuffered);
        });

        pdf.on("error", reject);

        pdf.end();
    });
}

sendPDF(docDefinition).then(pdfBuffer => {
    res.setHeader("Content-Type", "application/pdf");
    res.setHeader("Content-Length", pdfBuffer.length);
    res.send(pdfBuffer);
}).catch(err => {
    console.log(err);
    res.sendStatus(500);
});

Because there are many data events, you can't promisify just the data portion. You will still have to listen for each data event and collect the data.

Working with Node.js streams without callbacks

2 Answers2