Semantic Caching That Actually Saves Money

Introduction

A well-configured semantic cache is often the most effective way to reduce LLM API costs in production. If you are running an LLM-powered application, you have likely noticed that traditional caching strategies are bleeding money. Standard HTTP caching or database caching relies on exact string matching: if the incoming request string is identical to the cached key, you serve the result; otherwise, you pay for a new LLM call.

In the context of Large Language Models, exact matching is nearly useless. A user asking "What is the capital of France?" and another asking "Can you tell me the capital city of France?" are semantically identical but syntactically distinct. An exact-match cache misses both, forcing you to pay for two separate API calls for the same information.

This guide details the mechanism, implementation, and cost implications of semantic caching for platform engineers. By converting prompts into vector embeddings and comparing them in a high-dimensional space, we can identify when two different prompts ask the same question, significantly increasing cache hits and reducing API costs.

The Failure of Exact Caching

To understand the value of semantic caching, we must first quantify the failure mode of exact caching. LLM applications suffer from high variance in user phrasing. Users naturally employ synonyms, different sentence structures, and varying levels of formality. Because exact caching relies on byte-for-byte string equality, it cannot recognize that "How do I reset my password?" and "I forgot my login credentials, please help" refer to the same intent.

Consider a customer support bot. Users will phrase their issues in infinite ways:

1. "My password isn't working."
2. "I can't log in because my password failed."
3. "Help, I'm locked out of my account."

An exact-match cache generates a hash for each string. These hashes are unique. The cache hit rate for these three requests is 0%. If you have 10,000 unique but semantically similar queries, you pay for 10,000 LLM calls.

LLM APIs charge per token. A typical response might be 500 tokens. At $0.002 per 1k tokens (a conservative estimate for many models), that’s $0.001 per response. If you serve 1 million such responses, you spend $1,000. If semantic caching captures just 50% of these as "similar," you reduce the bill to $500. The mechanism is simple, but the implementation requires moving from string keys to vector keys.

The Mechanism: Embeddings and Similarity

Semantic caching operates in three distinct steps: embedding, storage, and retrieval. The core of this process revolves around managing a similarity threshold to balance accuracy with performance.

1. Embedding Generation

When a request arrives, the system does not look up the string. Instead, it passes the prompt through a small, fast embedding model (e.g., text-embedding-3-small from OpenAI or all-MiniLM-L6-v2). This model converts the text into a dense vector—a list of floating-point numbers—that represents the semantic meaning of the text.

# Conceptual example using OpenAI Python SDK v1
import openai
 
client = openai.OpenAI()
 
def get_embedding(text):
    response = client.embeddings.create(
        input=text,
        model="text-embedding-3-small"
    )
    return response.data[0].embedding

The resulting vector has 1536 dimensions (in this case). The distance between two vectors in this space correlates with their semantic similarity.

2. Vector Storage

These vectors are stored in a vector database or an in-memory store like FAISS, Pinecone, or Weaviate. Each entry in the cache includes:

• The original prompt (for verification)
• The embedding vector
• The LLM response
• Metadata (timestamp, model used)

3. Similarity Search and Thresholding

When a new request arrives, its embedding is generated and compared against all vectors in the cache. The system calculates a similarity score, typically using cosine similarity.

Cosine similarity measures the cosine of the angle between two vectors. It ranges from -1.0 (opposite) to 1.0 (identical). For text embeddings, scores usually range between 0.7 and 1.0 for relevant matches.

# Pseudo-code for similarity check
def check_semantic_cache(query_vector, cache_db):
    # Find the most similar vector in the cache
    results = cache_db.query(query_vector, top_k=1)
    
    if results:
        best_match = results[0]
        similarity_score = best_match.score
        
        # Define a threshold. 0.9 is strict, 0.7 is loose.
        if similarity_score >= SIMILARITY_THRESHOLD:
            return best_match.cached_response
    
    return None

The Critical Variable: Similarity Threshold

The SIMILARITY_THRESHOLD is your primary control knob. It determines the trade-off between precision and recall.

• High Threshold (e.g., 0.95): Only very close matches are served from cache. High precision (the cached answer is definitely correct), but lower recall (misses valid similar queries).
• Low Threshold (e.g., 0.80): Broader matches are served. Higher recall (more cache hits, more cost savings), but higher risk of serving an irrelevant answer (low precision).

For most use cases, a threshold between 0.85 and 0.92 provides a safe balance. You must tune this based on your specific domain. A legal advice bot requires a higher threshold than a creative writing assistant.

Handling Invalidation

Invalidation is the process of removing stale data from the cache to ensure users receive accurate information. Caches are only useful if the data is current. LLM outputs can be invalid if the underlying context changes. Semantic caching introduces a new dimension of invalidation: semantic drift.

Time-Based Invalidation (TTL)

The simplest strategy is Time-To-Live (TTL). If a cached response is older than, say, 24 hours, it is discarded. This is effective for static knowledge (e.g., "Who is the CEO of Apple?") but fails for dynamic data (e.g., "What is the stock price of AAPL?").

Semantic Invalidation

For dynamic data, TTL is insufficient. If the stock price changes, the embedding of the query "What is AAPL's price?" remains the same, but the answer is wrong.

Advanced semantic caching systems handle this by:

1. Contextual Tagging: Storing metadata about the source of truth. If the source (e.g., a database row) is updated, all cache entries referencing that source are invalidated.
2. Periodic Re-verification: For high-value queries, the system can periodically re-call the LLM to verify the cached answer against the current state, updating the cache if necessary.

The Consistency Trade-off

Platform engineers must decide: is the cost of an extra LLM call worth the guarantee of fresh data? For customer support, stale answers can lead to bad PR. For internal documentation search, a few hours of stale data is acceptable. Define your SLA, then configure your TTL and threshold accordingly.

Cost Savings: A Worked Scenario

Achieving significant cost savings requires understanding the math behind semantic caching. Let’s model a realistic scenario for a company processing 10,000 user queries per day.

Assumptions:

• Average prompt length: 50 tokens
• Average response length: 200 tokens
• Output cost: $0.002 per 1k output tokens
• Input cost: $0.0005 per 1k input tokens (Note: Input costs are included in the calculation below for accuracy, though they are often cheaper than output costs)
• Embedding cost: $0.00002 per 1k tokens (negligible compared to generation)

Scenario A: No Caching

• 10,000 requests
• 10,000 LLM calls
• Input Cost: 10,000 * 50 tokens * $0.0000005 = $0.25
• Output Cost: 10,000 * 200 tokens * $0.000002 = $4.00
• Total Cost: $4.25/day

Scenario B: Exact Match Caching

• Due to phrasing variations, only 5% of queries are identical strings.
• 500 hits, 9,500 misses.
• Input Cost: 10,000 * 50 * $0.0000005 = $0.25 (Input tokens are still processed for every query, regardless of cache hit or miss)
• Output Cost: 9,500 * 200 * $0.000002 = $3.80
• Total Cost: $4.05/day
• Savings: $0.20/day. Minimal.

Scenario C: Semantic Caching (Threshold 0.85)

• 60% of queries are semantically similar to previous ones.
• 6,000 hits, 4,000 misses.
• Input Cost: 10,000 * 50 * $0.0000005 = $0.25 (Embeddings generated for every query)
• Output Cost: 4,000 * 200 * $0.000002 = $1.60
• Total Cost: $1.85/day
• Savings: $2.40/day

Over a year, semantic caching saves approximately $876 for this modest workload. For a high-traffic application with 1 million daily requests, the savings scale to $87,600/year. This is not just optimization; it is a fundamental architectural shift that makes LLM applications economically viable at scale.

Implementation Checklist

Implementing effective prompt caching via semantic methods requires careful configuration. Follow this checklist to ensure robustness:

1. Choose an Embedding Model: Use a model optimized for semantic similarity (e.g., OpenAI text-embedding-3-small, Cohere embed-multilingual-v3). Avoid large LLMs for embedding; use dedicated, lightweight models.
2. Select a Vector Store: For small-scale apps, FAISS (in-memory) is sufficient. For distributed systems, use Pinecone, Weaviate, or pgvector.
3. Set Thresholds: Start with 0.85. Monitor false positives (incorrect answers served from cache). Adjust downward if you miss too many valid hits, or upward if you serve wrong answers.
4. Implement TTL: Set a default TTL based on your data freshness requirements.
5. Monitor Hit Rates: Track cache hit rates by semantic cluster. If hit rates are low, your threshold may be too high, or your embedding model may not be suited for your domain.

Conclusion

Semantic caching is not a silver bullet, but it is a necessary evolution for any production LLM system. By moving from string matching to vector similarity, you capture the vast majority of redundant queries that exact caching misses. The key to success lies in tuning the similarity threshold and managing invalidation strategically. The result is a system that is not only cheaper but also faster, as vector similarity searches operate in milliseconds compared to the seconds required for LLM generation.

Start small. Implement semantic caching for your most frequent, low-risk queries. Measure the impact. Then expand. The cost savings will compound, and the performance gains will be immediate.