Secrets Management for Python on AWS

I came from an application development background and during the early stage of my development journey, I had this constant struggle. How do I protect sensitive information like API keys in my application?

Hard coding said secrets in your application is a big NO-NO, especially when using version control tools such as Git since we may inevitably leak our secrets to the world. Keeping these secrets in the database is also not ideal since it subjects them to possible attack vectors such as SQL injections.

AWS Secrets Manager

In this article, we will be exploring the use of AWS Secrets Manager to store our sensitive data, and how we can use Python to retrieve these secrets.

To create a new secret in Secrets Manager, log on to your AWS console and search for Secrets Manager. In the Secrets Manager Console, click on “Store a new secret”

In the “Choose secret type” page, click on “Other type of secret” and key in the name and value of your secret in the “Key/value pairs” section. Under Encryption key, choose the AWS managed key, “aws/secretsmanager”. If you like, you could also choose a customer-managed key that you have created using KMS. Once you are done, click on “Next”.

Choose a secret type page

In the next screen, “Configure secret”, give your secret a name. By default the naming convention uses <environment>/<application name>/<application resource>. In this example, I am use “dev/likz/api” . Once you are done, click on “Next”.

Configure secret page

In the “Configure rotation — optional” page, make sure Automatic rotation is turned off. We are will not be using rotation in this session. Once you are done, click on the “Next” button

As the name suggests, automatic rotation isused to automatically rotate your secret based on a schedule. An AWS Lambda function is used to create a new secret. A quick overview of what’s happening behind the scene:

1. The rotation Lambda first calls Secrets Manager to retrieve a random password and stores the password in your current secrets with a PENDING state.
2. The rotation Lambda uses the current credentials to login to your service and sets the new password.
3. Once the password is set, the rotation Lambda then attempts to use the new password to login to your service.
4. If login is succesful, the rotation Lambda then set the PENDING password as CURRENT and all future request to AWS Secrets Manager will retrieve the new password.

At the bottom of the “Overview” page, AWS provides sample codes for the different programming languages. Click on “Python3”, copy and paste the code into an IDE or Notepady. Once you are done, click on “Store” to save the newly created secret.

Overview page

Setting up the AWS credentials

Make sure you have a valid set of AWS Credentials. To install the AWS CLI, refer to this link. To configure AWS CLI, refer here.

Make sure the user that you have created has the permission to read the secret from Secrets Manager.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::AccountId:role/EC2RoleToAccessSecrets"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*"
    }
  ]
}

If you used a Customer Manager key when storing your secret, make sure you also have the Decrypt permissions kms:Decrypt for the key.

Setting up the Python environment

Let’s set up the environment for running the Python code. Do note that for this demo I’m using a Windows setup.

First, make sure you have Python downloaded. You can download Python from the official Python website.

Create a new Python virtual environment (venv)

# Create a new virtual environment
python -m venv ./venv
# Active the new virtual environment
.\venv\Scripts\activate

Once you are in the virtual environment, download the boto3 SDK.

pip install boto3

Once the environment is set up, let’s edit the Python code we copied earlier to print out the secret (Note: DO NOT print out the secret for your actual application. We are printing it out as part of the demo)

At the bottom of the script, add the print(secret) code as follow

if 'SecretString' in get_secret_value_response:
    secret = get_secret_value_response['SecretString']
    print(secret) # Add here
else:
    decoded_binary_secret = base64.b64decode(get_secret_value_response['SecretBinary'])

Now, if we run the python code, we should be able to print the Secrets that we have created.

python main.py
{"TOKEN":"12345abcde"}

Conclusion

In this article, we have explored the use of Secrets Manager to protect sensitive information such as API keys using AWS Secrets Manager. AWS Secrets Manager not only provides a secure way of storing out secrets, it also provides automation capabilities to rotate the them.