How to save a file downloaded from S3 with Rusoto to my hard drive?

Question

I am trying to download a file from a bucket with Rusoto and I am getting the file content:

fn get_object(client: &TestClient, bucket: &str, filename: &str) {
    let get_req = GetObjectRequest {
        bucket: bucket.to_owned(),
        key: filename.to_owned(),
        ..Default::default()
    };

    let result = client.get_object(&get_req).sync().expect("Couldn't GET object");


    let stream = result.body.unwrap();
    let body = stream.concat2().wait().unwrap();

    assert!(body.len() > 0);
}

How can I save this GetObjectOutput(result) object to a file?

@hellow result variable type is GetObjectOutput which I shown in debugger — Akiner Alkan, Jul 11 '18 at 14:11

Shepmaster · Answer 1 · 2020-05-25T13:53:49.053

Rusoto now uses the standard library futures and no longer offers the sync method, so the previous answer is no longer valid.

Reading to memory

use futures::stream::TryStreamExt;
use rusoto_core::Region;
use rusoto_s3::{GetObjectRequest, S3Client, S3};

type Error = Box<dyn std::error::Error>;
type Result<T, E = Error> = std::result::Result<T, E>;

const BUCKET_NAME: &str = "my very own bucket name";

#[tokio::main]
async fn main() -> Result<()> {
    let client = S3Client::new(Region::UsEast2);

    let mut object = client
        .get_object(GetObjectRequest {
            bucket: BUCKET_NAME.into(),
            ..Default::default()
        })
        .await?;

    let body = object.body.take().expect("The object has no body");

    let body = body.map_ok(|b| b.to_vec()).try_concat().await?;
    println!("body length: {}", body.len());

    Ok(())
}

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY need to be specified. I chose to set environment variables outside of the code.

Streaming to a file

use rusoto_core::Region;
use rusoto_s3::{GetObjectRequest, S3Client, S3};
use tokio::{fs::File, io};

type Error = Box<dyn std::error::Error>;
type Result<T, E = Error> = std::result::Result<T, E>;

const BUCKET_NAME: &str = "my very own bucket name";

#[tokio::main]
async fn main() -> Result<()> {
    let client = S3Client::new(Region::UsEast2);

    let mut object = client
        .get_object(GetObjectRequest {
            bucket: BUCKET_NAME.into(),
            ..Default::default()
        })
        .await?;

    let body = object.body.take().expect("The object has no body");

    let mut body = body.into_async_read();
    let mut file = File::create("/tmp/a-place-to-write").await?;
    io::copy(&mut body, &mut file).await?;

    Ok(())
}

While ByteStream has an alluring into_blocking_read method, I do not recommend using it. If you attempt to use it inside of an async context, you get a panic because it starts a nested Tokio executor. If you use it outside of an async context, it will truncate the data unless you take great care to have the async runtime around but not to be within it.

Dependency versions

[dependencies]
rusoto_s3 = "0.43.0"
rusoto_core = "0.43.0"
tokio = { version = "0.2.21", features = ["macros"] }
futures = "0.3.5"

In this case should I pick your answer to note it is the correct one from now on? — Akiner Alkan, May 25 '20 at 15:34
@Websterix you should accept the answer that helped **you** the most, and upvote any answers that provide value. — Shepmaster, May 25 '20 at 15:35
Does anyone have an example which shows concurrent streams downloading (1:1) to files using the above? — Brian Bruggeman, Jan 10 '21 at 01:13
@BrianBruggeman see [How can I perform parallel asynchronous HTTP GET requests with reqwest?](https://stackoverflow.com/q/51044467/155423) for the general structure of such a thing. — Shepmaster, Jan 11 '21 at 14:11
@Shepmaster I think what you show here is good; does this imply that I can use multiple get object futures in place of urls? — Brian Bruggeman, Jan 11 '21 at 15:34
Couldn’t manage to put the async version to work, do you have a working version on GitHub or something? — Miguel Guilherme, Feb 22 '21 at 18:31

score 4 · Accepted Answer · answered Jul 12 '18 at 17:09

You're almost there. Your code will put the object in body, which is a Vec<u8>.

To write the contents of body to a file:

use std::io::Write;
use std::fs::File;

let mut file = File::create("/path/to/my-object").expect("create failed");
file.write_all(&body).expect("failed to write body");

How to save a file downloaded from S3 with Rusoto to my hard drive?

2 Answers2

Reading to memory

Streaming to a file

Dependency versions

Linked