How to ensure S3 prefixes are same when created from console vs CLI?

Question

I have the following folder structure that I would like to create in S3,

/demo/app/a.txt

Via Console:

demo and app using "Create Folder", followed by a.txt file upload

Via CLI:

aws s3 sync . s3://<my-bucket>/, where . (current dir) has demo folder

--

Now when I run,

aws s3 ls s3://<my-bucket> --recursive, the result is quite interesting/puzzling!

Output:

# created & uploaded from console
demo/
demo/app/
demo/app/a.txt

# from CLI
demo/app/a.txt

Clearly, there aren't any prefixes/objects for demo/ and demo/app/ created using CLI upload.

--

This answer helped me understand that "Create Folder" from console results in creating a 0-byte file when the folder is empty. But this isn't the case when uploaded via CLI.

So, how do I mimic the 0-byte file behavior for CLI uploads? In other words, match the prefixes!

The other use case is, navigating across directories from a browser will not work as the prefixes aren't available. For e.g.

1. Uploaded from console: https://<domain>/demo/ shows app directory.
2. Uploaded from CLI: https://<domain>/demo/ results in NoSuchKeyError

--

Note: I'm using a CloudFront distribution to access S3 data (if that helps)!

Answer 1

Amazon S3 is a flat object-storage system. It is not a filesystem and it does not have the concept of folders or directories. Rather, the Key (filename) of an object contains the full path of the object.

The easiest way to use S3 is to pretend that folders exist, but not actually create them. For example, you could copy a file to S3 like this:

aws s3 cp a.txt s3://my-bucket/demo/app/a.txt

This will work successfully, even if there is no directory called demo or app, because directories/folders do not exist.

Instead, Amazon S3 provides the concept of a CommonPrefix, which you'll see at the bottom of a ListObjects() API call. This returns a list of folder-like names that are separated by a delimiter (which is normally /). This provides programmatic equivalence to a directory, without actually needing them to exist.

If you wish to present a series of hierarchical directories, use the list of CommonPrefixes to build that view. This will work even when there are no zero-length files, because S3 looks at the Keys of the objects, not actual directories.

Here's some examples:

aws s3 cp a.txt s3://my-bucket/demo/app/a.txt
upload: ./a.txt to s3://my-bucket/demo/app/a.txt                  

aws s3api list-objects-v2 --bucket my-bucket

{
    "Contents": [
        {
            "Key": "demo/app/a.txt",
            "LastModified": "2020-04-22T01:11:20+00:00",
            "ETag": "\"802776735eb3ddcf03962ae47e08ed13\"",
            "Size": 211,
            "StorageClass": "STANDARD"
        }
    ]
}

aws s3api list-objects-v2 --bucket my-bucket --delimiter '/'

{
    "CommonPrefixes": [
        {
            "Prefix": "demo/"
        }
    ]
}

aws s3api list-objects-v2 --bucket jstack-b --delimiter '/' --prefix 'demo/'
{
    "CommonPrefixes": [
        {
            "Prefix": "demo/app/"
        }
    ]
}

Notice how the commands at the end provide a delimiter, so a list of CommonPrefixes are returned. This is a way that you could step through a list of directories (that don't exist).

Answer 2

If you really need to have a folder structure present in S3 that mirrors a local folder structure then you could try some variant of the following shell scripting and awscli on Linux or Mac:

1. list all local folders
2. transform that list into something matching S3 naming conventions
3. create a zero byte file at each folder name in S3

List local folders:

find . -type d

Example output:

.
./dogs
./dogs/cute
./dogs/fierce
./cats
./cats/white
./cats/black

Transform that list to remove . and so that ./dogs/cute becomes dogs/cute:

find . -type d | grep -v '^\.$' | sed 's/^\.\///g'

Example output:

dogs
dogs/cute
dogs/fierce
cats
cats/white
cats/black

Finally put it all together and create the zero byte file at each folder:

find . -type d \
    | grep -v '^\.$' \
    | sed 's/^\.\///g' \
    | xargs -L 1 -I % \
    aws s3api put-object --bucket mybucket --key %/ --content-length 0

Check what this resulted in in S3:

aws s3 ls s3://mybucket --recursive

Results:

2020-04-21 21:00:05          0 cats/black/
2020-04-21 21:00:05          0 cats/white/
2020-04-21 21:00:04          0 cats/
2020-04-21 21:00:03          0 dogs/cute/
2020-04-21 21:00:04          0 dogs/fierce/
2020-04-21 21:00:02          0 dogs/