API Keys: Rotation, Storage, and Monitoring

This is Part 5 of the Machine Identity and Workload Auth series.

API keys are the simplest form of machine identity: a static string sent in headers or query parameters to authenticate a client. While convenient, their static nature makes them high-value targets. Unlike short-lived OAuth tokens, API keys often have longer lifespans and may not auto-expire unless configured to do so, though many providers enforce rotation policies. For backend and platform engineers, managing this lifecycle is not just a security checkbox—it is an operational necessity.

The Mechanism of Risk

The primary danger of an API key lies in its persistence. A typical API key does not expire automatically, creating a large "blast radius." If a key is leaked—through a git commit, a log file, or a memory dump—an attacker can impersonate the service indefinitely. Tools like git-secrets can help prevent accidental commits, but prevention is better than detection.

Consider a backend service that calls a third-party payment provider. The service uses an API key with write permissions. If that key is exposed, an attacker can initiate fraudulent transactions. Because the key is static, the attacker can retry requests from different IP addresses, mimicking legitimate traffic patterns. The only mitigation is detection and revocation, which introduces latency between compromise and containment.

Secure Storage Patterns

Where you store an API key determines how easily it can be exfiltrated. The hierarchy of storage security is strict:

1. Code Repositories: Never store API keys in source code. Version control systems (Git, SVN) are not designed for secret management. Even if you use .gitignore, accidental commits happen. Tools like git-secrets or pre-commit hooks can scan for patterns, but prevention is better than detection.
2. Environment Variables: Acceptable for local development and simple deployments. However, environment variables are often exposed to all processes running on a host, increasing the risk of leakage via process listing or core dumps.
3. Secret Managers: The gold standard for production. Services like AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault provide encrypted storage, access logging, and automated rotation. These services use short-lived credentials (IAM) for access, ensuring the secret value is only retrieved on-demand and not hardcoded.

When retrieving secrets, always use the provider's SDK or CLI within your application startup sequence. Do not hardcode retrieval commands. For example, in a Kubernetes environment, use CSI drivers or external-secrets operators to inject secrets as volumes or environment variables, ensuring they are never written to disk in plaintext if possible.

Rotation Strategies

Rotation is the process of replacing an old API key with a new one. The goal is to minimize the window of opportunity for an attacker using a compromised key. There are two primary strategies: manual and automated.

Manual Rotation

Manual rotation involves generating a new key, updating the configuration, and deprecating the old one. This is error-prone because it requires human coordination. If the update fails, the service may go down. If the old key is deleted before the new one is fully propagated, you experience downtime. Furthermore, manual updates are susceptible to human error, such as typos in configuration files or forgetting to update all instances in a cluster, leading to inconsistent states where some services use the old key and others the new.

Automated Rotation with Grace Periods

Automated rotation integrates with CI/CD pipelines. The process typically follows these steps:

1. Generate a new API key.
2. Store the new key in the secret manager.
3. Deploy the updated configuration to the application.
4. Wait for a grace period (e.g., 24 hours) where both old and new keys are valid.
5. Revoke the old key.

This grace period is critical. It ensures that any in-flight requests or cached tokens using the old key do not fail. During this window, monitoring systems should track usage of both keys. If the old key shows no activity after the grace period, it can be safely revoked.

Example of a rotation script in a CI/CD pipeline:

# Generate new key via provider CLI
NEW_KEY=$(aws secretsmanager get-random-password --exclude-punctuation --length 64 --query RandomPassword --output text)
 
# Update secret in AWS Secrets Manager
aws secretsmanager put-secret-value \
  --secret-id my-api-key \
  --secret-string "$NEW_KEY"
 
# Trigger deployment
kubectl rollout restart deployment/my-service
 
# Monitor for 24 hours, then revoke old key
sleep 86400
# Note: Manual revocation is an application-side action or use cancel-rotation if stopping rotation.
# aws secretsmanager rotate-secret --secret-id my-api-key

Monitoring & Anomaly Detection

Storage and rotation are preventive controls. Monitoring is detective. You must know when an API key is being used abnormally. Key behaviors to monitor include:

• Geolocation Anomalies: A key issued in us-east-1 suddenly appearing in requests from eu-west-1 within minutes.
• Endpoint Access: A key that only accesses /v1/users suddenly attempting to access /admin/config.
• Volume Spikes: A sudden increase in request rate, indicating automated scraping or brute force attacks.
• Error Rates: A high rate of 401/403 errors may indicate an attacker trying to guess key validity or using a revoked key.

Set up alerts for these anomalies. Use tools like Prometheus, Datadog, or cloud-native monitoring services to track API key usage. Tag your metrics with the key ID to enable granular analysis.

Opinion: Avoid alerting on every failed request. This leads to alert fatigue. Instead, alert on thresholds (e.g., >100 failures per minute) or specific high-risk actions (e.g., access to admin endpoints).

Conclusion

API key management is not a one-time setup. It is a continuous cycle of generation, storage, rotation, and monitoring. By treating API keys as sensitive credentials rather than simple configuration values, backend and platform engineers can significantly reduce their attack surface. Implement automated rotation, use secret managers, and monitor for anomalies. These steps transform API keys from a liability into a manageable component of your security posture.

Common Pitfalls

• Storing keys in client-side code: Exposing API keys in frontend JavaScript or mobile apps allows anyone to view the source and steal credentials. Always use backend proxies.
• Ignoring key expiration policies: Assuming keys are permanent increases risk. Configure expiration dates and enforce rotation policies to limit the window of exposure.
• Failing to monitor for anomalies: Without monitoring, you won't know if a key has been compromised until significant damage is done. Set up alerts for unusual traffic patterns and error rates.

Practical Takeaways

• Principle of Least Privilege: Grant each API key only the permissions it strictly needs. Avoid using admin-level keys for routine tasks.
• Automate Everything: Manual processes introduce human error. Use CI/CD pipelines and secret managers to automate key generation, rotation, and revocation.
• Monitor Continuously: Treat API key usage like any other security metric. Regularly review logs and set up alerts for suspicious activity.

FAQ

How often should I rotate API keys? Rotate API keys at least every 90 days, or sooner if a compromise is suspected. Automated rotation tools can make this process seamless and frequent.

Can I use environment variables in production? Environment variables are acceptable for simple deployments but are not recommended for production due to security risks. Use dedicated secret managers like AWS Secrets Manager or HashiCorp Vault for better security and management.

What should I do if I suspect an API key is compromised? Immediately revoke the key. Then, investigate the logs to determine the extent of the compromise and rotate the key. Update all services that use the key with a new one.