Copying a file from S3 into my codebase when using Elastic Beanstalk

Question

I have the following script:

Parameters:
  bucket:
    Type: CommaDelimitedList
    Description: "Name of the Amazon S3 bucket that contains your file"
    Default: "my-bucket"
  fileuri:
    Type: String
    Description: "Path to the file in S3"
    Default: "https://my-bucket.s3.eu-west-2.amazonaws.com/oauth-private.key"
  authrole:
    Type: String
    Description: "Role with permissions to download the file from Amazon S3"
    Default: "aws-elasticbeanstalk-ec2-role"

files:
  /var/app/current/storage/oauth-private.key:
    mode: "000600"
    owner: webapp
    group: webapp
    source: { "Ref" : "fileuri" }
    authentication: S3AccessCred

Resources:
  AWSEBAutoScalingGroup:
    Type: "AWS::AutoScaling::AutoScalingGroup"
    Metadata:
      AWS::CloudFormation::Authentication:
        S3AccessCred:
          type: "S3"
          roleName: { "Ref" : "authrole" }
          buckets: { "Ref" : "bucket" }

The issue that I am having is that when this is being deployed, the file aren't present in the /var/app/current/storage directory.

I thought that maybe this script was running too soon and the current directory wasn't ready yet, so I tried the ondeck directory and this also doesn't work.

If I change the path to anywhere other than my codebase directory it works, the file is copied from S3.

Any ideas? Thanks.

littleforest · Accepted Answer · 2020-04-14T20:40:49.197

2

Directives under the "files" key are processed before your web application is set up. You will need to download the file to a tmp file, and then use a container_command to transfer it to your app in the current directory.

This AWS doc mentions near the top the order in which keys are processed. The files key is processed before commands, and commands are run before the application and web server are set up. However, the container_commands section notes that they are used "to execute commands that affect your application source code".

So you should modify your script to something like this:

Parameters: ...
Resources: ...

files:
  "/tmp/oauth-private.key":
    mode: "000600"
    owner: webapp
    group: webapp
    source: { "Ref" : "fileuri" }
    authentication: S3AccessCred

container_commands:
  file_transfer_1:
    command: "mkdir -p storage"
  file_transfer_2:
    command: "mv /tmp/oauth-private.key storage"

edited Apr 14 '20 at 20:40

answered Apr 14 '20 at 01:28

littleforest

2,057
21
29

This is perfect. Did the trick. I pretty sure you need to be a robot to understand and navigate through the AWS docs. So many contradicting articles. Although, just one amendment. `file_transfer_3` is not required as `mv` deletes it already. – ChrisBratherton Apr 14 '20 at 17:13
Ah yes, I'll fix that. I think I was copying from one of my examples that used `cp` for some reason. Glad this helped! – littleforest Apr 14 '20 at 20:40
Spot on! Thanks again! – ChrisBratherton Apr 15 '20 at 08:32
Works for deployment, but an error on the ElasticBeanstalk configuration update. From eb-engine.log `[INFO] Error occurred during build: Command file_transfer_2 failed` – Yuriy Matviyuk Nov 15 '22 at 11:24

Copying a file from S3 into my codebase when using Elastic Beanstalk

1 Answers1

Linked