0

I need to process some files in an S3 bucket using hive. Problem is, hive doesn't apparently like using the buckets as folders, so I decided to create a folder in another bucket, and copy the files there via s3cmd.

This is my command:

s3cmd -v cp s3://main.bucket/\* s3://temp.bucket/input/

s3cmd then promptly declares I have X files to transfer (usually around the order of 5k++) and then after copying one file, conks out on me with the following error:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please report the following lines to:
   s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Problem: KeyError: 'dest_name'
S3cmd:   1.0.1

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 2006, in <module>
    main()
  File "/usr/local/bin/s3cmd", line 1950, in main
    cmd_func(args)
  File "/usr/local/bin/s3cmd", line 614, in cmd_cp
    subcmd_cp_mv(args, s3.object_copy, "copy", "File %(src)s copied to %(dst)s")
  File "/usr/local/bin/s3cmd", line 604, in subcmd_cp_mv
    dst_uri = S3Uri(item['dest_name'])
KeyError: 'dest_name'

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
    Please report the above lines to:
   s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Now, that just sucks.

Thinking that the problem lies in bucket-bucket transferring, I decided to create a folder inside the main.bucket, and simply copy everything in there, uncaring if the folder everything is to be copied into is also copied, thus causing a loop or whatnot, because at that point I was at my wit's end.

The new command is the following:

s3cmd -v cp s3://main.bucket/\* s3://main.bucket/111a/

Again, it copied but one file, and gave me the same error.

What could this mean? How do I fix this?

Thanks.

P.S. If all else fails, I suppose I could do it via AWS's GUI, but manually checking every checkbox is a pain. To my knowledge, if you scroll down long enough, it will stupidly forget the files you checked above, forcing you to do it at a snail's pace.

If there's a method to select them all at once, and copy paste them into a new directory, that would be great.

zack_falcon
  • 1,033

1 Answers1

1

You are using a very old version of s3cmd. I believe this bug has been fixed in the years since. Please try again with upstream github.com/s3tools/s3cmd master branch. As of today, 1.5.0-rc1 is the most current release.

Matt Domsch
  • 111