Downloading from S3 with aws-cli using filter on specific prefix

Question

For some reason there's a bucket with a bunch of different files, all of which have the same prefix but with different dates:

backup.2017-01-01aa

backup.2017-01-01ab

backup.2017-01-15aa

backup.2017-01-15ab

backup.2017-02-01aa

backup.2017-02-01ab

etc..

How do I download only files that start with "backup.2017-01-01"?

lol, someone [downvoted without adding a comment](https://imgur.com/3LeRBy0), thanks! That's really helpful and helps me improve my question/answer a lot! /s — ZN13, Jul 28 '17 at 23:03

Sampo Smolander · Accepted Answer · 2021-10-20T11:25:44.707

31

I think --include does the filtering locally. So if your bucket contains millions of files, the command can take hours to run, because it needs to download a list of all the filenames in the bucket. Also, some extra network traffic.

But aws s3 ls can take a truncated filename to list all the corresponding files, without any extra traffic. So you can

aws s3 ls s3://yourbucket/backup.2017-

to see your files, and something like

aws s3 ls s3://yourbucket/backup.2017- | colrm 1 31 | xargs -I % aws s3 cp s3://yourbucket/% .

to copy your files.

edited Oct 20 '21 at 11:25

answered Mar 25 '19 at 16:03

Sampo Smolander

1,605
2
15
33

3

This must be the selected answer! – juliangonzalez Sep 26 '19 at 16:23
no, because it *neither* does not work fast enough. – Antti Haapala -- Слава Україні Dec 12 '19 at 08:23
3

this works great. way faster than the selected answer – Sonic Motion Apr 08 '20 at 17:21
1

If the prefix in ls command has "/" separator, make sure to specify the prefix up to the separator in the cp command (this is necessary as ls returns "file" names, while cp requires full path including the "directories") – Yaegor Mar 23 '21 at 14:28
2

Finally a good answer! This is great if you have a lot of files in your bucket. – vincent May 07 '21 at 22:29
2

@AnttiHaapala--СлаваУкраїні if you have lot of small files, adding `-P 15` to `xargs` will run 15 files in parallel, significantly speeding up downloading of large number of files. – Lukáš Doležal Mar 29 '22 at 12:33

score 28 · Answer 2 · edited Apr 11 '19 at 08:07

You'll have to use aws s3 sync s3://yourbucket/

There are two parameters you can give to aws s3 sync; --exclude and --include, both of which can take the "*" wildcard.

First we'll have to --exclude "*" to exclude all of the files, and then we'll --include "backup.2017-01-01*" to include all the files we want with the specific prefix. Obviously you can change the include around so you could also do something like --include "*-01-01*".

That's it, here's the full command:

aws s3 sync s3://yourbucket/ . --exclude "*" --include "backup.2017-01-01*"

Also, remember to use --dryrun to test your command and avoid downloading all files in the bucket.

score 1 · Answer 3 · answered Jun 27 '22 at 16:44

A PowerShell equivalent for @sampo-smolander's answer

,@(Get-ChildItem -recurse | aws s3 ls s3://yourbucket/backup.2017-) | Select-Object -ExpandProperty syncroot |  foreach-Object {$_.split(" ")[-1]} | %{&"aws" s3 cp s3://yourbucket/$_ .}

Posting it here since I spent a lot of time figuring out this, so hopefully it helps someone else who needs to use powershell. Also I'm not too familiar with powershell so it may need optimization.

Downloading from S3 with aws-cli using filter on specific prefix

3 Answers3

Linked