Use the aws client to copy s3 files from a single directory only (non recursively)

Question

Consider an aws bucket/key structure along these lines

 myBucket/dir1/file1
 myBucket/dir1/file2
 myBucket/dir1/dir2/dir2file1
 myBucket/dir1/dir2/dir2file2

When using:

 aws s3 cp  --recursive s3://myBucket/dir1/ .

Then we will copy down dir2file[1,2] along with file[1,2]. How to only copy the latter files and not files under subdirectories ?

Responding to a comment: . I am not interested in putting a --exclude for every subdirectory so this is not a duplicate of excluding directories from aws cp

Possible duplicate of https://stackoverflow.com/questions/39879330/excluding-directories-from-aws-cp — Abhishek Garg, Jul 16 '19 at 17:20
Just to make sure, I understood the question correctly. Let's say there are more files. myBucket/dir1/dir3/dir2file1 myBucket/dir1/dir3/dir2file2 do you want to copy these files also or only from dir2? — Abhishek Garg, Jul 16 '19 at 17:22
only files directly under `dir1` and not under subdirectories — WestCoastProjects, Jul 16 '19 at 17:52
@AbhishekGarg This is *not* a duplicate of that question. I am not interested in putting a `--excludes` for *every* subdirectory — WestCoastProjects, Jul 16 '19 at 17:53
Yes, that's what I thought. Thanks for clarifying. Added an answer. — Abhishek Garg, Jul 16 '19 at 18:37

matsev · Answer 1 · 2019-07-16T17:16:54.460

1

You can exclude paths using the --exclude option, e.g.

aws s3 cp s3://myBucket/dir1/ . --recursive --exclude "dir1/dir2/*"

More options and examples can be found by using the aws cli help

aws s3 cp help

edited Jul 16 '19 at 17:16

answered Jul 16 '19 at 17:12

matsev

32,104
16
121
156

The asterisk is not expanded by the shell locally? – WestCoastProjects Jul 16 '19 at 17:16
This does not meet the needs of the question .. I don't want to put an `--excludes` for _every_ subdirectory – WestCoastProjects Jul 16 '19 at 17:54

Abhishek Garg · Accepted Answer · 2019-07-16T18:57:32.657

1

As far as I understood, you want to make sure that the files present in current directories are copied but anything in child directories should not be copied. I think you can use something like that.

aws s3 cp s3://myBucket/dir1/ . --recursive --exclude "*/*"

Here we are excluding files which will have a path separator after "dir1".

edited Jul 16 '19 at 18:57

answered Jul 16 '19 at 18:37

Abhishek Garg

2,158
1
16
30

Well you get precisely the opposite of what I was looking for ;) But looks like we can simply invert the conditions and would "fly". Can you make that change? I'll upvote in the meantime – WestCoastProjects Jul 16 '19 at 18:49
I am slightly embarrassed that I didn't understand the question correctly. Updated the answer. – Abhishek Garg Jul 16 '19 at 18:58
does not copy anything ;( – WestCoastProjects Jul 16 '19 at 19:01
I tried this aws s3 ls s3://mapbox-test.xxxx.com/ --recursive --profile nq 2019-07-16 11:56:51 137191 dir1/abc.jpg 2019-07-10 14:32:52 831 index.html aws s3 cp s3://mapbox-test.xxx.com/ . --recursive --exclude "\*/\*" This was the output. download: s3://mapbox-test.xxx.com/index.html to ./index.html – Abhishek Garg Jul 16 '19 at 19:07
The comment format is making it confusing. But basically, there were 2 files and with the command, it only copied the file in current directory. – Abhishek Garg Jul 16 '19 at 19:09

score 0 · Answer 3 · answered Jul 16 '19 at 19:29

There is no way you can control the recursion depth while copying files using aws s3 cp. Neither it is supported in aws s3 ls.

So, if you do not wish to use --exclude or --include options, I suggest you:

Use aws s3 ls command without --recursive option to list files directly under a directory, extract only the file names from the output and save the names to a file. Refer this post
Then write a simple script to read the file names and for each execute aws s3 cp

Alternatively, you may use:

aws s3 cp s3://spaces/dir1/ .  --recursive --exclude "*/*"

Use the aws client to copy s3 files from a single directory only (non recursively)

3 Answers3