How to copy from subdirectories using s3DistCp

Question

Trying to use s3DistCp to copy from s3://my-bucket/dir1/ , s3://my-bucket/dir2, s3://my-bucket/dir3.

And all three DIRs has some files in them. Wanted to do something like:

hadoop jar s3distcp.jar --src s3://my-bucket/*/ --dest s3://my-bucket/some-other-dir/

But it generate an error saying that:

's3://my-bucket/*/' directory not found...

So does it mean s3DistCp doesn't take wildcards in paths? Is there any work around or any ideas?

how about writing a bash script that simply calls s3distcp multiple times with each of the source paths? — Tal Joffe, Sep 22 '16 at 06:42

score 0 · Answer 1 · edited May 23 '17 at 12:19

0

edited May 23 '17 at 12:19

Community

answered Aug 02 '14 at 09:31

samthebest

No... I removed the trailing '/' but it is still generating "no such file or directory" error. – yunt Aug 04 '14 at 16:22

1 Answers1