AWS s3 V3 Javascript SDK stream file from bucket (GetObjectCommand)

Question

Iv looked all over AWS docks and stack overflow (even went to page 4 of google!!!) but i cannot for the life of me work out how to stream a file from S3. The docs for V3 are pretty useless and all the examples i find are from V2.

The send commond that V3 uses only returns a promise so how do i get a stream and pipe it instead of waiting for the whole file (it needs to be piped into encryption algo then to a response stream)

this.s3.send(
  new GetObjectCommand({
    Bucket: '...',
    Key: key,
  }),
);

I was able to upload fine by passing the stream as the body, is there something i have to do similar here?

  uploadToAws(key) {
    const pass = new PassThrough();

    return {
      writeStream: pass,
      promise: this.s3.send(
        new PutObjectCommand({
          Bucket: '...',
          Key: key,
          Body: pass,
          ServerSideEncryption: '...',
          ContentLength: 37,
        }),
      ),
    };
  }

Answer 1

Body from the GetObjectCommand is a readable stream (https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/getobjectcommandoutput.html#body).

So you can do:

const command = new GetObjectCommand({
    Bucket
    Key,
});
const item = await s3Client.send(command);
item.Body.pipe(createWriteStream(fileName));

Answer 2

Using transformToString() as per docs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_GetObject_section.html.

export const handler = async event => {
    try {
            
        // Retrieve the object from S3
        const data = await s3.getObject({ Bucket: BUCKET_NAME, Key: PATH_AND_FILE_NAME });
    
        // Set the content type of the response
        const contentType = data.ContentType;

        // Convert to base64 string
        const streamToString = await data.Body?.transformToString("base64");
    
        // Return the object data in the response
        return {
            statusCode: 200,
            headers: {
                "Content-Type": contentType
            },
            body: streamToString,
            isBase64Encoded: true
        };
    } catch (error) {           
        return {
            statusCode: 500,
            body: "An error occurred: " + error.message
        };
    }
};

Answer 3

Figured it out,

s3Client.send(command) returns a type GetObjectCommandOutput.
const data: GetObjectCommandOutput = await s3Client.send(command)

data.Body is of type SdkStream<Readable | ReadableStream<any> | Blob | undefined> | undefined

Undefined is for error cases, you check there is no error case like this

if (!data.Body)
//handle error

For success case, you can get ReadableStream like this
const readableStream: ReadableStream = data.Body!.transformToWebStream()

For aws-sdk V2, there was createReadStream(), this is seems to be the way in v3.

To pipe through ReadableStream, use
readableStream.pipeTo() or readableStream.pipeThrough()

Answer 4

docs links

For those landing here after googling because aws v3 sdk docs are missing details on getobjectcommandoutput interface, you can find the full getobjectcommandoutput definition at source or at "module" → https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/modules/getobjectoutput.html

Answer 5

In case you also run into a similar issue while using typescript, This helped me

(object.Body as any).pipe(res);

I did this because as mentioned by laconbass, aws v3 sdk docs are missing details on getobjectcommandoutput.

Answer 6

Try this in TypeScript for streaming to a file path, destinationPath:

import { createWriteStream } from "node:fs";
import { GetObjectCommand, S3Client } from "@aws-sdk/client-s3";
import { Readable } from "node:stream";

const s3 = new S3Client({});

async function downloadFile(
  bucketName: string,
  objectKey: string,
  destinationPath: string
) {
  const data = await s3.send(
    new GetObjectCommand({ Bucket: bucketName, Key: objectKey })
  );
  return new Promise(async (resolve, reject) => {
    const body = data.Body;
    if (body instanceof Readable) {
      const writeStream = createWriteStream(destinationPath);
      body
        .pipe(writeStream)
        .on("error", (err) => reject(err))
        .on("close", () => resolve(null));
    }
  });
}

Answer 7

To get a blob instead of writing into a file, you can just do this:

const command = new GetObjectCommand({
   Bucket
   Key,
});
const { Body } = await s3Client.send(command);
const blob = await Body.transformToByteArray();

Answer 8

This is what worked for me -

const fileStream = fs.createReadStream('/tmp/myfile.ext');

const command = new PutObjectCommand({ Body: fileStream, Bucket, Key });

const result = await client.send(command);

if(result?.$metadata?.httpStatusCode === 200){
    console.log('Success');
};