Caching Strategies for RAG Pipelines

Retrieval-Augmented Generation (RAG) pipelines are fundamentally bottlenecked by two expensive operations: generating vector embeddings for static knowledge bases and performing similarity searches for every user query. Without caching, these operations repeat identically for repeated inputs, wasting compute cycles and increasing latency. Effective caching in RAG is not a single monolithic layer but a dual-strategy system: an Embedding Cache that stores pre-computed vector representations of documents, and a Retrieval Cache that stores query vectors and their corresponding result sets.

The core challenge is not storage, but invalidation. RAG systems often operate on dynamic data. If the underlying knowledge base changes, cached embeddings become stale, and cached retrieval results become incorrect. This article details the mechanism-level design of these caches, how they interact, and how to manage invalidation without sacrificing performance.

The Cost of Redundancy

A standard RAG pipeline involves three steps: ingestion (chunking and embedding documents), retrieval (embedding the user query and searching the vector database), and generation (prompting the LLM).

The ingestion phase is write-heavy and batch-oriented. The retrieval phase is read-heavy and request-driven.

Without caching, every user query triggers a new embedding call to the embedding model. For large batches of similar queries (e.g., "What is the return policy?" asked by 100 users), this results in identical computational work. Furthermore, during ingestion, if a document is updated, the entire vector store may need to be rebuilt or updated, a process that is computationally expensive and often blocks other operations.

Caching breaks this cycle by storing the results of expensive computations. However, unlike traditional web caching where content is static, RAG data is often semi-static or dynamic. This requires a cache that understands semantic relationships and data versioning.

Embedding Cache: Storing Pre-Computed Vectors

The Embedding Cache sits at the ingestion layer. Its purpose is to avoid regenerating embeddings for documents that have not changed since the last processing cycle.

Mechanism: Hash-Based Indexing

The embedding cache does not store raw text. It stores the mapping between a document's unique identifier (or content hash) and its vector representation.

1. Input: A document chunk D with content C.
2. Hashing: Compute a content hash H = SHA256(C).
3. Lookup: Check if H exists in the cache.
4. Hit: Return the existing vector V.
5. Miss: Call the embedding model to generate V, store (H, V) in the cache, and return V.

This approach ensures that if the same text appears in multiple chunks or is re-ingested after a system restart, the embedding model is not called.

Implementation Details

In practice, the embedding cache is often implemented as a key-value store (like Redis) or a separate table in the vector database itself.

import hashlib
import numpy as np
 
def get_embedding(text: str, cache: dict, embedder: Model):
    # Create a deterministic hash of the content
    content_hash = hashlib.sha256(text.encode('utf-8')).hexdigest()
    
    if content_hash in cache:
        return cache[content_hash]  # Cache Hit
    
    # Generate embedding (Cache Miss)
    vector = embedder.encode(text)
    cache[content_hash] = vector
    return vector

The key insight here is that the cache key is based on content, not on the document ID. This allows the cache to handle deduplication automatically. If two documents have identical content, they share the same embedding, saving memory and compute.

Retrieval Cache: Storing Query Results

The Retrieval Cache operates at the query layer. It stores the mapping between a user's query vector (or the query text itself) and the top-k results returned by the vector database.

Exact Match vs. Semantic Similarity

A naive retrieval cache uses exact match on the query vector. This is simple but brittle. If a user asks "How do I reset my password?" and another asks "How can I change my password?", the vectors will be different, resulting in a cache miss despite the semantic intent being identical.

To improve hit rate, advanced retrieval caches use semantic caching. This involves:

1. Storing the query vector and the result set.
2. On a new query, compute its vector.
3. Search the cache for the nearest neighbor to the new query vector.
4. If the distance between the new query vector and the cached query vector is below a threshold θ, return the cached results.

This introduces a trade-off: higher hit rates but potential for irrelevant results if the threshold is too loose.

Invalidation Challenges

The retrieval cache is more volatile than the embedding cache. Every time a document is added, updated, or deleted, the top-k results for all cached queries may change. Invalidating the entire retrieval cache on every write is expensive and reduces hit rates.

A common pattern is to use a TTL (Time-To-Live) for retrieval cache entries. However, this is imprecise. A better approach is to tag cache entries with the version of the knowledge base. When the knowledge base is updated, increment the version number and invalidate all entries associated with the previous version.

Invalidation Strategies: Balancing Consistency and Performance

Invalidation is the hardest part of RAG caching. There are three primary strategies:

1. Content-Based Invalidation (Embedding Cache)

For the embedding cache, invalidation is straightforward. If a document's content changes, its hash changes. The old hash no longer matches, so the cache naturally serves the new embedding. This is a "lazy" invalidation strategy. The old entry remains in the cache until it is evicted by memory pressure or TTL. This is acceptable because the cost of regenerating the embedding is low compared to the cost of searching the vector store.

2. Version-Based Invalidation (Retrieval Cache)

For the retrieval cache, maintain a global version counter for the knowledge base. Each cache entry stores the version it was generated against. On query time, if the current knowledge base version differs from the cached entry's version, treat it as a miss. This ensures consistency without requiring a full cache flush on every write.

3. TTL-Based Invalidation (Hybrid)

Use TTLs for both caches as a safety net. Set the embedding cache TTL to a long duration (e.g., 24 hours) since document content rarely changes. Set the retrieval cache TTL to a shorter duration (e.g., 1 hour) since query patterns may shift.

Worked Scenario: Hybrid Cache Layer

Consider a customer support RAG pipeline. The knowledge base is updated nightly, but user queries come in bursts.

1. Ingestion:

   • Documents are hashed.
   • Embeddings are generated and stored in a Redis cache keyed by content hash.
   • The vector database is updated with the new embeddings.
   • Knowledge base version is incremented to v2.

2. Query Time:

   • User asks: "What is the API rate limit?"
   • Query vector is computed.
   • Check Retrieval Cache for exact match or semantic neighbor within θ=0.1.
   • Hit: Return cached top-5 documents. Latency: <10ms.
   • Miss: Search vector database. Retrieve top-5 documents. Store query vector and results in Retrieval Cache with version v2. Latency: ~200ms.

3. Update Time:

   • A document is updated.
   • Embedding cache: Old hash evicted, new hash added.
   • Retrieval cache: All entries with version v2 are marked invalid. New queries will miss and populate v3 entries.

This hybrid approach ensures that the most expensive operation (vector search) is cached, while the less expensive operation (embedding generation) is also optimized. The version-based invalidation ensures that users always see up-to-date information, even if the cache is warm. Note that in this context, the cache entry stores a snapshot of the global version at the time of caching, preventing implementation ambiguity between the global counter and the local tag.

Conclusion

Caching in RAG pipelines is not a one-size-fits-all solution. The Embedding Cache optimizes ingestion by deduplicating content via hashing. The Retrieval Cache optimizes query latency by storing results, enhanced by semantic similarity matching. The key to success is a clear invalidation strategy that combines content-based hashing for embeddings and version-based tracking for retrieval results. By implementing these layers, you can reduce latency by an order of magnitude in high-traffic scenarios and significantly lower the cost of LLM and vector database usage.