How to check AWS secretsmanager rotation is completed successfully

Question

Created a secret in AWS secretsmanager, enabled automatic rotation with lambda function. when I trigger rotation for the first time from cli, It's not completed. This is the initial state of secret when updated secret in aws console manually.

# aws secretsmanager list-secret-version-ids --secret-id ******
{
    "Versions": [
        {
            "VersionId": "9e82b9e2-d074-478e-83a5-baf4e578cb49",
            "VersionStages": [
                "AWSCURRENT"
            ],
            "LastAccessedDate": 1592870400.0,
            "CreatedDate": 1592889913.431
        },
        {
            "VersionId": "e32ddaf8-7f21-40e2-adf8-f976b8f3f104",
            "VersionStages": [
                "AWSPREVIOUS"
            ],
            "LastAccessedDate": 1592870400.0,
            "CreatedDate": 1592887518.46
        }
    ],
    "ARN": "arn:aws:secretsmanager:us-east-1:***********:secret:***********",
    "Name": "*******"
}

Now I triggered rotation from aws cli

aws secretsmanager rotate-secret --secret-id ******

# aws secretsmanager list-secret-version-ids --secret-id ********
{
    "Versions": [
        {
            "VersionId": "704102f3-b36d-4529-b257-0457354d3c93",
            "VersionStages": [
                "AWSPENDING"
            ],
            "CreatedDate": 1592890351.334
        },
        {
            "VersionId": "e32ddaf8-7f21-40e2-adf8-f976b8f3f104",
            "VersionStages": [
                "AWSPREVIOUS"
            ],
            "LastAccessedDate": 1592870400.0,
            "CreatedDate": 1592887518.46
        },
        {
            "VersionId": "9e82b9e2-d074-478e-83a5-baf4e578cb49",
            "VersionStages": [
                "AWSCURRENT"
            ],
            "LastAccessedDate": 1592870400.0,
            "CreatedDate": 1592889913.431
        }
    ],
    "ARN": "arn:aws:secretsmanager:us-east-1:**********:secret:********",
    "Name": "********"
}

Cloudwatch log stopped at this createSecret: Successfully put secret for ARN arn:aws:secretsmanager:xxxxxxx.. looks like only createsecret function is called. When I rotate the secret again, Gets this output in cli

An error occurred (InvalidRequestException) when calling the RotateSecret operation: A previous rotation isn't complete. That rotation will be reattempted.

Unable to understand what's happening. Can someone help?

Marcin · Answer 1 · 2020-06-23T06:33:37.577

2

Unfortunately there is no out-of-the-box way for that, as Secrets Manger does not have build in SNS notification nor CloudWatch Events for when rotation completes.

Thus, you have to construct a solution yourself, which can be done using SDK or CLI.

For CLI you can use describe-secret and pull secret details in a loop. In the loop, you have to look into AWSPENDING and AWSCURRENT labels for the versions.

From the docs:

If instead the AWSPENDING staging label is present but is not attached to the same version as AWSCURRENT then any later invocation of RotateSecret assumes that a previous rotation request is still in progress and returns an error.

So basically, looking at your output:

        {
            "VersionId": "704102f3-b36d-4529-b257-0457354d3c93",
            "VersionStages": [
                "AWSPENDING"
            ],
            "CreatedDate": 1592890351.334
        }

you have a version with AWSPENDING label, which is not attached to the same version as AWSCURRENT. This indicates that the rotation is in progress.

The rotation completes, when a version is in one of the two states:

The AWSPENDING and AWSCURRENT staging labels are attached to the same version of the secret, or The AWSPENDING staging label is not attached to any version of the secret.

edited Jun 23 '20 at 06:33

answered Jun 23 '20 at 06:23

Marcin

215,873
14
235
294

I have the code which moves AWSCURRENT label to version Id of AWSPENDING in fnishSecret step. Following this template https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/blob/master/SecretsManagerRotationTemplate/lambda_function.py I"m wondering why finisheSecret step not called at all. This question https://stackoverflow.com/questions/60007180/how-to-verify-the-secrets-manager-credential-rotation-is-successful also seem to be of same issue. Any idea why only createSecret step called when rotated? – user9163519 Jun 23 '20 at 06:47
@user9163519 I'm not familiar with that template. Sorry. – Marcin Jun 23 '20 at 07:00
1

@Marcin Can you elaborate the solution a bit? Do i have to remove AWSPENDING? Will rotation complete on its own or i have to do something? – Rohit Mittal Sep 16 '21 at 07:01
2

@RohitMittal I would suggest making new question specific to your use-case with details that are relevant to your requirements, errors, etc. – Marcin Sep 16 '21 at 07:04
https://stackoverflow.com/questions/69203935/an-error-occurred-keypairsecretrotationschedule-received-when-doing-serverless – Rohit Mittal Sep 16 '21 at 07:17

score 2 · Answer 2 · answered Jun 23 '20 at 18:09

Secrets Manager will publish an event via CloudTrail - 'RotationSucceeded' when there is a successful rotation.

See this for more information on how to setup a Cloudwatch alarm off that CloudTrail event - https://docs.aws.amazon.com/secretsmanager/latest/userguide/monitoring.html

How to check AWS secretsmanager rotation is completed successfully

2 Answers2