216

I'm having trouble using * in the AWS CLI to select a subset of files from a certain bucket.

Adding * to the path like this does not seem to work:

aws s3 cp s3://data/2016-08* .

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
metersk
  • 11,803
  • 21
  • 63
  • 100
  • 7
    This question is currently [being discussed on Meta](https://meta.stackoverflow.com/q/423124/8239061). – SecretAgentMan Feb 09 '23 at 18:25
  • 3
    Please don't attempt to close or reopen this question while it's discussed on meta, it's no use, as people will disagree with the outcome and will close/reopen it again. Better use your limited amount of votes for all the complete garbage we receive daily. – Lino Feb 09 '23 at 20:26

4 Answers4

382

To download multiple files from an Amazon AWS bucket to your current directory, you can use the recursive, exclude, and include flags. The order of the parameters matters.

Example command:

aws s3 cp s3://data/ . --recursive --exclude "*" --include "2016-08*"

For more information on how to use these filters: Use of Exclude and Include Filters

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
punkrockpolly
  • 9,270
  • 6
  • 36
  • 37
  • 76
    I'd like to point out the `--exclude "*"` isn't a typo. *If you don't add it, the include will match anything*. As per the documentation: Note that, by default, all files are included. This means that providing only an --include filter will not change what files are transferred. --include will only re-include files that have been excluded from an --exclude filter. If you only want to upload files with a particular extension, you need to first exclude all files, then re-include the files with the particular extension. – pyb Jan 28 '19 at 22:49
  • 18
    Be aware of the order of exclude and include as well. – keiki May 27 '19 at 12:27
  • 7
    You can also use `sync` for a similar effect, which is recursive by default: `aws s3 sync s3://data/ . --exclude "*" --include "2016-08*"` – enharmonic May 29 '20 at 21:18
  • 8
    It is a good idea to test "destructive" commands with the `--dryrun` flag to make sure the correct set of object is selected by the wildcard. – nerfologist Oct 09 '20 at 10:09
  • Also good to note that the `--recursive` flag is required. Or it will not work. – Akaisteph7 Jun 27 '22 at 15:36
84

The Order of the Parameters Matters

The exclude and include should be used in a specific order, We have to first exclude and then include. The vice versa of it will not be successful.

aws s3 cp s3://data/ . --recursive  --include "2016-08*" --exclude "*"

This will fail because order of the parameters maters in this case. The include is excluded by the *.

aws s3 cp s3://data/ . --recursive --exclude "*" --include "2016-08*"`

This one will work because then we excluded everything, but later we had included the specific directory.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
loneStar
  • 3,780
  • 23
  • 40
11

Okay I have to say the example is wrong and should be corrected as follows:

aws s3 cp . s3://data/ --recursive --exclude "*" --include "2006-08*" --exclude "*/*"

The . needs to be right after the cp. The final --exclude is to make sure that nothing is picked up from any subdirectories that are picked up by the --recursive (learned that one by mistake...)

This will work for anyone struggling with this by the time they got here.

Arturo Moncada-Torres
  • 1,236
  • 14
  • 23
Patrick Palmer
  • 135
  • 1
  • 3
  • 12
    this is for copying _from_ your current local current directory _to_ the specified S3 location. all the other responses is copying _from_ s3 location _to_ your local current directory – Dexter Legaspi Nov 13 '20 at 00:53
-3

If there is an error while using ‘ * ’ you can also use recursive, include, and exclude flags like

aws s3 cp s3://myfiles/ . --recursive --exclude "*" --include "file*"

MD. RAKIB HASAN
  • 3,670
  • 4
  • 22
  • 35
Peters Monday
  • 1
  • 1