2

I've got an ASG that assigns an IAM Role to each of the instances that join it. Therefore, each instance has the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables baked-in, which will be used upon instantiation to download and decrypt credentials that are stored in an S3 bucket and encrypted using KMS keys.

So I'll have the following components:

  • An S3 bucket called top-secret.myapp.com
  • All objects in this bucket are encrypted using a KMS key called My-KMS-Key
  • An IAM instance role with inline policies attached granting it the ability to interact with both the bucket and the KMS key used to encrypt/decrypt the contents of the bucket (see below)
  • A user data script that installs the aws-cli upon instantiation and then goes about attempting to download and decrypt an object from the top-secret.myapp.com bucket.

The User Data Script

Upon instantiation, any given instance runs the following script:

#!/bin/bash

apt-get update
apt-get -y install python-pip
apt-get -y install awscli

cd /home/ubuntu
aws s3 cp s3://top-secret.myapp.com/secrets.sh . --region us-east-1
chmod +x secrets.sh
. secrets.sh
shred -u -z -n 27 secrets.sh


IAM Role Policies

The IAM role for my ASG instances has three policies attached inline:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::top-secret.myapp.com"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*"
            ],
            "Resource": [
                "arn:aws:s3:::top-secret.myapp.com/secrets.sh"
            ]
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:*"
            ],
            "Resource": "arn:aws:kms:us-east-1:UUID-OF-MY-SECRET-KEY-HERE"
        }
    ]
}

The first policy is essentially a full-root-access policy with no restrictions. Or so I thought, but it doesn't work. So I thought it might be that I need to explicitly apply policies that allow interaction with S3 encryption and/or KMS, makes sense.

So I added the second policy that allows the IAM instance role to list the top-secret.myapp.com bucket, and LIST and GET the secrets.sh object within the bucket. But this produced the error illustrated below.

The (Unknown) Error I'm Getting

download failed: s3://top-secret.myapp.com/secrets.sh to ./secrets.sh
A client error (Unknown) occurred when calling the GetObject operation: Unknown

Anyone have any idea what could be causing this error?

Note: This method for transferring encrypted secrets from S3 and decrypting them on-instance works fine using the standard Amazon S3 service master key

AJB
  • 7,389
  • 14
  • 57
  • 88
  • So, it seems that your problem is specifically concerning the "aws s3 cp" command, and doesn't (at the moment) involve any problems with KMS. The error message doesn't suggest that there is a permission-related problem. Are you able to copy any other files (eg from a public S3 bucket)? Have you tried the `--debug` option for the AWS CLI? – John Rotenstein May 03 '15 at 10:33
  • Thanks for the suggestion, John. But it's definitely `KMS` that's the issue here. It works fine with the default `S3 SSE` key, just not with any keys I assign via `KMS`. It must be something to do with the policies attached to any given resource, but I struggle to understand those. I'll have to get back to this later this week. – AJB May 05 '15 at 01:48

1 Answers1

2

For me, the issue was two-fold:

  1. If you're using server-side encryption via KMS, you need to supply the --sse aws:kms flag to the aws s3 cp [...] command.
  2. I was installing an out-of-date version of awscli (version 1.2.9) via apt, and that version didn't recognize the --sse aws:kms command
    • Running apt-get remove awscli and installing via pip install awscli gave me version 1.10.51, which worked.

EDIT:

If you're using a different KMS key than the default master key for your account, you will need to also add the following flag:

--sse-kms-key-id [YOUR KMS KEY ID]

wilkystyle
  • 327
  • 1
  • 11