Why the Secrets Manager Bill Surprises Teams

This is Part 5 of the AWS Secrets and Key Management series.

Many teams underestimate the aws secrets manager expensive reality, assuming storage is the primary driver of secrets manager cost. AWS Secrets Manager violates this intuition by operating on a transactional model where every API call is a billable event. For teams managing hundreds of secrets in high-throughput applications, this distinction is the primary driver of unexpected cost spikes. The mechanism is simple but brutal: you are charged per GetSecretValue request, regardless of payload size.

The Transactional Mechanism

AWS Secrets Manager pricing is divided into two components: storage per secret per month and API requests. The storage cost is modest, typically around $0.40 per secret per month. The API cost, however, scales linearly with traffic. As of current pricing, you pay approximately $0.05 per 10,000 API requests (regardless of operation type) see AWS Secrets Manager pricing.

This model surprises teams because they view secrets as static configuration. In reality, each retrieval is a network hop to an AWS control plane endpoint, followed by cryptographic decryption and access control checks. If your application architecture involves thousands of microservices, each polling for credentials independently, the API call volume compounds rapidly. The bill does not reflect the value of the secret, but the frequency of its access. From my experience auditing infrastructure, a common "surprise bill" scenario involves a team migrating a monolith to microservices without adjusting their secret retrieval logic; the number of secrets remains low, but the number of calls explodes, leading to a bill that is 50x higher than the previous database connection pool costs.

The Caching Failure Mode

The most common source of cost explosion is the absence of client-side caching. Many legacy applications or poorly configured frameworks retrieve secrets on every single HTTP request or database connection attempt. Consider a web application serving 10,000 requests per second. If each request requires a fresh secret lookup (e.g., for a database credential), that is 864 million API calls per day.

# Anti-pattern: Direct API call on every request
def get_db_connection():
    # Expensive: Hits AWS Secrets Manager API every time
    response = secrets_client.get_secret_value(SecretId='my-db-secret')
    secret = json.loads(response['SecretString'])
    return connect_to_database(secret['username'], secret['password'])

In this scenario, the application generates over 864 million GetSecretValue calls daily. At $0.05 per 10,000 requests, this single function would incur approximately $129,600 in API costs per month. This is not an edge case; it is a common misconfiguration in high-concurrency environments where developers assume the SDK handles caching automatically. It does not. The AWS SDK for Python (Boto3), Java, or Go is inherently synchronous. To achieve asynchronous behavior, developers must use separate async libraries (e.g., aioboto3) or async wrappers, not a configuration option on the standard client see Boto3 documentation.

The Consolidation Anti-Pattern

Beyond raw volume, the structure of your secrets matters. Teams often create individual secrets for every environment, region, or service instance (e.g., prod/api-service/db, staging/api-service/db, prod/web-service/db). While this improves isolation, it fragments access patterns. If you have 1,000 distinct secrets, each accessed once per second, you still face 86.4 million calls daily. However, consolidation can reduce overhead if multiple services share the same credential.

Consolidation is not just about reducing the number of secrets; it is about reducing the fan-out of API calls. If five microservices share a single database credential, they should share the cached value, not make five independent API calls for the same data. AWS recommends grouping secrets by access pattern rather than strictly by environment if cost is a concern, though security boundaries must be respected.

Mitigation via Caching and TTL

The solution is client-side caching with a defined Time-To-Live (TTL). Since secrets rarely rotate more than once every few hours or days, caching the decrypted value locally eliminates redundant API calls. The key is balancing cost reduction with security requirements for rotation.

A typical TTL of 5 to 10 minutes is sufficient for most use cases. This reduces API calls by orders of magnitude. For example, if you cache the secret for 5 minutes, a service making 10,000 requests per second will only make one API call every 300 seconds — roughly 0.0033 calls per second — regardless of the underlying request volume. This drastically lowers the API call rate compared to the uncached scenario.

import boto3
import json
import time
 
class SecretCache:
    def __init__(self, ttl_seconds=300):
        self.secrets_manager = boto3.client('secretsmanager')
        self.cache = {}
        self.ttl = ttl_seconds
 
    def get_secret(self, secret_id):
        now = time.time()
        if secret_id in self.cache:
            secret_data, cached_time = self.cache[secret_id]
            if now - cached_time < self.ttl:
                return secret_data
        
        # Cache miss: Fetch from AWS
        response = self.secrets_manager.get_secret_value(SecretId=secret_id)
        secret_string = response['SecretString']
        secret_data = json.loads(secret_string)
        
        # Update cache
        self.cache[secret_id] = (secret_data, now)
        return secret_data

This wrapper ensures that even under heavy load, the number of API calls is proportional to the number of unique secrets and the TTL, not the number of requests. For the previous example of 10,000 requests per second with a 5-minute TTL, the API call rate drops from millions to roughly one call every 5 minutes per logical service instance.

Conclusion

The cost of AWS Secrets Manager is not a function of the secrets themselves, but of the access pattern. Teams that treat secret retrieval as a low-cost, high-frequency operation will face exponential bills. By implementing client-side caching with appropriate TTLs and consolidating access where security permits, engineers can reduce API call volume by 99% or more. Always audit your GetSecretValue metrics in CloudWatch. If you see high request counts, you have a caching problem, not a storage problem.

Common Pitfalls

Lambda Cold Starts: Forgetting that Lambda functions may not have a persistent cache across cold starts, leading to a burst of API calls at the start of execution. Use provisioned concurrency or externalize the cache if possible.
High-Frequency Tokens: Using Secrets Manager for short-lived tokens that change every few seconds. The API call overhead will likely exceed the cost of generating the token locally or using a faster, specialized service.
IAM Policy Overhead: Ignoring the fact that every API call also triggers IAM policy evaluation. While not directly billed in the same way, excessive API calls increase the load on IAM and can lead to throttling.

Practical Takeaways

Implement client-side caching: Use a TTL-based cache (e.g., 5-10 minutes) to reduce API calls by orders of magnitude.
Consolidate secret access patterns: Group secrets by shared access rather than strict environmental isolation where security permits.
Monitor CloudWatch API call metrics: Set up alerts for high GetSecretValue request counts to catch caching failures early.