AWS Secrets Manager Multi-User Rotation

Introduction

In this article, we will look at a multi-user secrets rotation strategy and why we need it. We will not be covering code in this article, but we will look at secrets rotation at a high level to better understand how it works.

For those who are technical, AWS publishes a repository of sample secrets rotation code that can be found here.

AWS Secrets Manager

The AWS Secrets Manager has built-in capabilities to help rotate stored sensitive information, such as database credentials. Rotating secrets is a best practice that organisations often use to minimise the likelihood of leaked credentials and limits the impact of a leaked secret since they are time-bound.

When rotating credentials with AWS Secrets Manager, an AWS Lambda function is called to orchestrate the steps needed to rotate the secret.

One thing that most may miss is that AWS Secrets Manager calls AWS Lambda a total of four times.

1. The first invocation is made with a step name called createSecret. As the name suggests, this step is used by Lambda to call AWS Secrets Manager to generate a new password and is stored back in Secrets Manager in an AWS_PENDING state.
   Note: Only secrets in the AWS_CURRENT state are retrieved by applications when they call getSecret on AWS Secrets Manager.
2. The second execution is called setSecret, and is used to update your service with the new set of credentials.
3. The third is called testSecret, which tests the secrets stored with the AWS_PENDING state against the new credentials set in Step 2.
4. Lastly, the finishSecret step moves the secret from the AWS_PENDING state to the AWS_CURRENT state.

Below is a sequence diagram that outlines the steps.

This rotation sequence is often called a “Single User Rotation”.

Limitations

The limitation of the Single User rotation implementation is that it creates a period where the credentials stored in AWS Secrets Manager and the actual credentials in Amazon RDS are out-of-sync. This is shown in the diagram below.

During the area marked in red, if a user or service calls AWS Secrets Manager for the credentials to the database, an outdated set of credentials will be returned. The password will be invalid if the user uses the credentials to log in to the database.

Solution: Multi-User Rotation

The multi-user rotation Lambda strategy uses a similar implementation as the single-user rotation, except it creates and updates a clone of the user during a rotation instead of the active one.

This means there are always two active set of credentials at all times. Below is a diagram that shows how this is being done. The steps that have changed have been bolded for ease of reading.

In this implementation, instead of creating a new password, the Lambda function also creates a clone of the user during the first rotation. For subsequent rotations, the original and clone users will be alternated for each rotation. The new password is then saved along with the clone user in the AWS_PENDING state.

In the setSecret stage, a new clone user is created instead of updating the current user for the first rotation. Similarly, subsequent rotations will alternate between the two.

Below is another diagram of what the rotation would look like after the first rotation.

With this implementation, one important thing to note is that your rotation schedule must be halved. For example, if your organisation requires a secret to be rotated every 90 days, the rotation schedule must be set to 45 days. This is because, for every rotation, the alternate user is being updated.

1. Original: dbuser, password1 (Day 0)
2. First rotation: dbuser_clone, password2 (Day 45)
3. Second rotation: dbuser, password3 (Day 90)
4. Third rotation: dbuser_clone, password4 (Day 135)

Another benefit of maintaining two active sets of credentials means applications can cache the password up to half the duration of your rotation policy. In the above example, this means that as long as the application refreshes the credentials by calling AWS Secrets Manager once every 45 days, the credentials will remain valid.

Conclusion

The multi-user rotation is another strategy for rotating your secrets to minimise application downtime. However, this only works if the service you are integrating allows you to maintain two or more active credentials. For services that do not, the single-user implementation must be used.