CloudFront returning 403 instead of 404 for missing pages from S3 using OAC

Question

Using AWS CloudFormation I set up a CloudFront distribution to serve content from a private S3 bucket. I do not have the bucket configured as an S3 website — rather, I'm using the latest-and-greatest technique: Origin Access Control (OAC). See Restricting access to an Amazon S3 origin. I'm using Route53 and Certificate Manager to serve the CloudFront distribution over TLS with a custom domain example.com.

So far the basics are working fine for URLs that reference objects that exist in the S3 bucket. I can access https://example.com/foobar.html just fine, for example. But if I request a file that does not exist, such as https://example.com/missing.html, CloudFront returns a 403 "Access Denied" instead of a 404 "Not Found".

I can make a wild guess that some communication between CloudFront and S3 makes CloudFront think its access is denied if the object doesn't exist. (Still that doesn't explain why.) Is this a bug? Is this expected behavior? How are we expected to use CloudFront+S3+OAC with this odd behavior—does AWS expect us to set up a CloudFront custom error response to convert 403 to 404? (But why would we want to assume all access denied errors in CloudFormation really indicate a missing object on S3?)

Note that I found various other CloudFront questions related to 403, but none related to an OAC configuration, and most of the other questions were regarding a CloudFront distribution that always returned 403, not just for missing files.

Answer 1

Unless you have the s3:ListBucket permission, S3 returns the 403 Forbidden status and the AccessDenied error for missing objects, by design. This is because without s3:ListBucket, the principal doesn't have permission to know whether the object is missing or if it exists but they aren't allowed access.

Note that unlike s3:GetObject, an object-level permission where the resource ARN is arn:aws:s3:::bucket-name/*, s3:ListBucket is a bucket-level permission, so the resource is arn:aws:s3:::bucket-name without the trailing /*.

After updating the bucket policy, you should find that the 404s work as expected, but you also need to set the Cloudfront Default Root Object for the distribution to whatever you want returned when / is requested, otherwise a bucket listing will be returned, which is probably not what you want.

Also be aware of the Error Caching Minimum TTL, which causes CloudFront to cache those 403s for 5 minutes, separate from the other TTL settings for the cache behavior.

Answer 2

Michael's answer is exactly correct. I'm leaving a separate answer here just to flesh out the details of exactly what needs to be done. I'll use CloudFormation for illustration.

Here's an example of updating the S3 bucket policy to include s3:GetObject:

  BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref Bucket
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: cloudfront.amazonaws.com
            Action: s3:ListBucket
            Resource: !Sub "${Bucket.Arn}"
            Condition:
              StringEquals:
                AWS:SourceArn: !Sub "arn:aws:cloudfront::${AWS::AccountId}:distribution/${CloudFrontDistribution}"
          - Effect: Allow
            Principal:
              Service: cloudfront.amazonaws.com
            Action: s3:GetObject
            Resource: !Sub "${Bucket.Arn}/*"
            Condition:
              StringEquals:
                AWS:SourceArn: !Sub "arn:aws:cloudfront::${AWS::AccountId}:distribution/${CloudFrontDistribution}"

As Michael mentioned, you need to set DefaultRootObject to something or CloudFront will return a directory listing when / is requested. But you still need to indicate a default page if any sub-path root is indicated, such as /foo/bar/. And what if you don't want to indicate a default page? You can address both issues using CloudFront functions.

First set up a parameter for the default page:

Parameters:
  DefaultPage:
    Description: The default page, such as `index.html`.
    Type: String
    Default: index.html

Then set up the function:

  DefaultPageFunction:
    Type: AWS::CloudFront::Function
    Properties:
      Name: default-page
      AutoPublish: true
      FunctionConfig:
        Comment: Convert root requests to default page.
        Runtime: cloudfront-js-1.0
      FunctionCode: !Sub |
        function handler(event) {
          var request = event.request;
          var uri = request.uri;
          if (uri.endsWith("/")) {
            var defaultPage = "${DefaultPage}";
            if(defaultPage) {
              request.uri += defaultPage;
            } else {
              return {
                statusCode: 403,
                statusDescription: "Forbidden"
              };
            }
          }
          return request;
        }

This function handles both cases based upon the DefaultPage parameter. If it's set to a non-empty string, it will add that to any …/ path. If DefaultPage is empty, then it will return an HTTP 403 Forbidden for every …/ path.

Finally add it to your AWS::CloudFront::Distribution DefaultCacheBehavior:

          …
          FunctionAssociations:
            - EventType: viewer-request
              FunctionARN: !GetAtt DefaultPageFunction.FunctionMetadata.FunctionARN
        DefaultRootObject: !Ref DefaultPage

I threw in the DefaultRootObject for good measure, even though the function should take care of that case as well.

Warning! There seems to be a bug with CloudFormation that will get the parameter "stuck" with one value, so double-check that CloudFormation deployed the correct value if you modify DefaultPage after deploying. See the ticket I just opened: CloudFormation deploy doesn't update !Sub contents with new parameter value. #989