How to save file from S3 using aws-sdk v3

Question

With the new aws-sdk v3, specifically @aws-sdk/client-s3, it's not clear to me how to store files to disk from S3.
Having the following code

const { Body } = await s3.send(new GetObjectCommand({
  Bucket: 'myBucket',
  Key: 'myKey',
}))

The Body can be of type Readable | ReadableStream | Blob.
I'd like to store this into a file on disk. Since the return types are completely different, is there an easy way to e.g. get a Buffer or Stream.Readable from aws-sdk?

In aws-sdk v2 it was easy with .createReadStream() function which seems to no longer exist.

Check this issue https://github.com/aws/aws-sdk-js-v3/issues/1096#issuecomment-620900466 — Alex Po, Mar 18 '21 at 13:34

score 15 · Accepted Answer · edited Feb 08 '22 at 19:54

15

It appears ReadableStream | Blob is available only in browser. Node.js always gets Readable which is what I need.

The solution is pretty straightforward afterwards with fs.createWriteStream and piping the Readable such as

await new Promise((resolve, reject) => {
  Body.pipe(fs.createWriteStream(filePath))
    .on('error', err => reject(err))
    .on('close', () => resolve())
})

For reference, here's an issue where improvement to documentation is discussed https://github.com/aws/aws-sdk-js-v3/issues/1877

edited Feb 08 '22 at 19:54

nornagon

15,393
18
71
85

answered Mar 18 '21 at 17:34

mike

5,047
2
26
32

If you're using Typescript - see the enhancement from Abigail Nguyen below. – Joseph Siefers Mar 27 '23 at 15:17

score 2 · Answer 2 · answered Oct 09 '21 at 10:17

2

To do this in typescript you can also do

await new Promise((resolve, reject) => {
 if (Body instanceof Readable) {
   Body.pipe(fs.createWriteStream(filePath))
    .on('error', err => reject(err))
    .on('close', () => resolve())
 }
});

answered Oct 09 '21 at 10:17

Abigail Nguyen

129
1
6

1

Note that you may have to ```import { Readable } from 'stream'``` – Joseph Siefers Mar 27 '23 at 15:16

SRINIVASA YADAV TALAPANTI · Answer 3 · 2022-07-14T07:18:59.913

GetObjectCommand returns readable stream, we use fs module to create writeStream and pass the returned readableStream from GetObjectCommand to writeStream to save the file

const fs = require('fs');
const { S3Client, GetObjectCommand } = require('@aws-sdk/client-s3');

const s3Client = new S3Client({
  region: 'your_region',
  credentials: { accessKeyId: 'your_accessKeyId', secretAccessKey: 'your_secretAccessKey' },
});

const bucketParams = {
  Bucket: 'your_bucket',
  Key: 'your_key.jpg',
};

const run = async () => {
  try {
    const data = await s3Client.send(new GetObjectCommand(bucketParams));
    const inputStream = data.Body;
    const downloadPath = 'example.png';
    const outputStream = fs.createWriteStream(downloadPath);
    inputStream.pipe(outputStream);
    outputStream.on('finish', () => {
      console.log(`downloaded the file successfully`);
    });
  } catch (err) {
    console.log('Error', err);
  }
};

score 1 · Answer 4 · answered May 05 '23 at 21:32

You can use the streaming methods, as mentioned in the responses above, but you can also just use the writeFile method from the fs/promises library:

const { writeFile } = require("node:fs/promises");
const { S3Client, GetObjectCommand } = require("@aws-sdk/client-s3");

const downloadFile = async () => {
    const command = new GetObjectCommand({ Bucket, Key });
    const { Body } = await client.send(command);

    const localFilename = "path_to_where_to_save_file";

    await writeFile(localFilename, Body);
}

and then call this function.

I think this is from node version v14.18.0 onwards, because this method accepts a stream as input.

How to save file from S3 using aws-sdk v3

4 Answers4

Linked