Automatic Secret Rotation With Lambda

Managing static database credentials is a primary vector for security breaches. When credentials are hardcoded or manually updated, they tend to persist long after they should be changed. AWS Secrets Manager solves this by treating secrets as mutable resources with versioned states. However, the security benefit only materializes if the rotation happens automatically. This is achieved by attaching an AWS Lambda function to the secret resource, turning the rotation process into a managed workflow rather than a manual script.

The core mechanism relies on a state machine defined by VersionStage labels. Secrets Manager does not simply overwrite the secret value. Instead, it maintains two logical pointers within the secret version history: AWSCURRENT and AWSPENDING. The application always reads from AWSCURRENT. During rotation, the Lambda function writes new credentials to AWSPENDING. Only after the new credentials are verified against the database does Secrets Manager swap the labels, making the new credentials AWSCURRENT and demoting the old ones to AWSPREVIOUS. This ensures that any application reading the secret during the rotation window continues to authenticate successfully, preventing downtime.

The Lambda Invocation Contract

Secrets Manager does not call a generic function; it invokes a Lambda function with a specific event payload that dictates the current phase of rotation. The Lambda function must implement four distinct handlers to satisfy the rotation API contract. These handlers are identified by the Event field in the invocation payload.

1. createSecret: This handler is called when the secret has no AWSPENDING version or when the previous rotation failed. Its job is to generate new credentials (e.g., a new database user) and write them to the secret under the AWSPENDING stage.
2. setSecret: This handler is called after createSecret succeeds. It instructs the Lambda to propagate the new credentials stored in AWSPENDING to the actual database or service. For RDS, this often means creating the new IAM user or updating the password in the database directly.
3. testSecret: Before committing the change, Secrets Manager needs assurance that the new credentials work. This handler attempts to authenticate with the database using the AWSPENDING credentials. If authentication fails, the rotation is aborted, and the secret remains unchanged.
4. finishSecret: This is the commit step. Upon success, this handler signals completion to Secrets Manager, which then performs the atomic label swap from AWSPENDING to AWSCURRENT.

# Note: These are stubs. In practice, the Lambda must parse the 'Event' field 
# from the 'event' argument to route logic to the correct handler (createSecret, 
# setSecret, testSecret, or finishSecret).
 
import json
import boto3
import os
 
def create_secret(event, context):
    # Logic to create new DB user and write to AWSPENDING
    pass
 
def set_secret(event, context):
    # Logic to update the new user's password in the DB
    pass
 
def test_secret(event, context):
    # Logic to verify AWSPENDING credentials can log in
    pass
 
def finish_secret(event, context):
    # Logic to signal success to Secrets Manager for label swap
    pass

The Alternating User Strategy

For database credentials, the most common and effective rotation strategy is "Alternating Users." This approach avoids the complexity of changing passwords for an existing user while connections are active, which is impossible without dropping sessions. Instead, the rotation lambda manages two users: the current active user and a new pending user.

When rotation is triggered, the Lambda first checks the secret’s metadata. If there is no AWSPENDING version, it executes createSecret. In this step, the Lambda connects to the database using the credentials currently in AWSCURRENT. It creates a new database user, say db_user_new, and generates a random password. This password is then stored in Secrets Manager under the AWSPENDING version stage. The original user, db_user_old, remains active and unchanged in the database.

Next, Secrets Manager invokes setSecret. The Lambda retrieves the AWSPENDING credentials (containing db_user_new and its password). It then grants db_user_new the exact same permissions as db_user_old. This step is critical; if permissions are not mirrored, the application will fail authentication even with the correct password. At this point, the database has two users with identical privileges, but only db_user_old is known to the running applications via AWSCURRENT.

The testSecret handler then attempts to connect to the database using db_user_new. If the connection succeeds, it proves that db_user_new exists, has the correct password, and possesses sufficient permissions. If this test fails, the Lambda returns an error, and Secrets Manager does not proceed to finishSecret. The AWSPENDING version is retained, allowing for retry or manual inspection.

Finally, finishSecret signals completion to Secrets Manager. Secrets Manager then performs the atomic label swap, moving the AWSCURRENT label from the old version (containing db_user_old) to the new version (containing db_user_new). Simultaneously, it moves the old version to AWSPREVIOUS. Once this label swap is complete, any subsequent read of the secret by an application will return the credentials for db_user_new. The old credentials for db_user_old are now effectively deprecated, though they remain in the version history for audit purposes.

Configuration and Dependencies

Implementing secrets manager rotation requires configuring the secret in Secrets Manager with a rotation rule. You specify the Lambda ARN as the rotation function and define the rotation interval (e.g., every 30 days). Secrets Manager also requires specific IAM permissions. The Lambda function must have permission to read and write the secret (secretsmanager:GetSecretValue, secretsmanager:PutSecretValue, secretsmanager:UpdateSecretVersionStage) and invoke itself if retries are needed (lambda:InvokeFunction). Additionally, the Lambda needs permissions to interact with the target service, such as rds-db:connect for RDS instances.

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:GetSecretValue",
    "secretsmanager:PutSecretValue",
    "secretsmanager:UpdateSecretVersionStage",
    "lambda:InvokeFunction"
  ],
  "Resource": "arn:aws:secretsmanager:region:account:secret:my-secret-*"
}

This architecture shifts the burden of security from human discipline to system enforcement. By leveraging the VersionStage state machine and the alternating user pattern, you ensure that credentials are always fresh, never exposed in code, and rotated without impacting application availability. The Lambda acts as the orchestrator, but the security guarantee comes from the immutable version history within Secrets Manager, providing a clear audit trail of every credential change.

Conclusion

Automatic secret rotation transforms credential management from a manual, error-prone task into a reliable, auditable system process. By defining the VersionStage lifecycle and implementing the four-part Lambda contract, you can ensure that database credentials are rotated without downtime. The alternating user strategy provides an effective pattern for handling concurrent access, while strict IAM policies ensure that only authorized functions can modify secret states. Implementing this setup reduces the attack surface significantly, keeping your infrastructure secure against compromised credentials.