Hybrid Search: Combining BM25 and Dense Retrieval

In modern search systems, relying on a single retrieval mechanism creates a predictable blind spot. Lexical search engines like Elasticsearch or Solr excel at exact keyword matching but fail when users query using synonyms or natural language variations. Conversely, dense vector retrieval (semantic search) captures intent and synonymy but struggles with exact term matching, negation, and rare proper nouns. Hybrid search is not merely a feature flag; it is an architectural pattern that combines BM25 (Best Matching 25) and dense retrieval to cover these gaps, often alongside sparse models like splade. This post explains the mechanism of fusion, specifically Reciprocal Rank Fusion (RRF), and how to implement it in a production RAG pipeline.

Part 5 of the RAG from Zero to Production series.

The Divergence Mechanism

To understand why hybrid search is necessary, we must look at the failure modes of individual retrievers.

BM25 operates on term frequency-inverse document frequency (TF-IDF) principles. It calculates a score based on how often a query term appears in a document relative to its frequency in the corpus.

• Strength: Exact match. If a user searches for "ERROR 500", BM25 will prioritize documents containing that exact string.
• Weakness: Semantic disconnect. A document discussing "HTTP status code 500" or "server failure" will have a low BM25 score for the query "ERROR 500" because the tokens do not overlap, despite high semantic relevance.

Dense Retrieval uses embedding models (e.g., BERT, E5) to map text into a high-dimensional vector space. Similarity is measured by cosine distance.

• Strength: Semantic recall. The vector for "server failure" will be close to "ERROR 500" in the embedding space.
• Weakness: Lack of precision. The model may conflate distinct concepts. For example, the query "Java" might retrieve documents about "JavaScript" or "coffee" if the embedding space is not tightly constrained, because the vector representations can overlap in broad semantic clusters. Additionally, dense models often ignore negation. A query like "dog NOT cat" might still retrieve documents mentioning both, as the vector for the whole phrase represents an average of the concepts.

Hybrid search leverages the complementary nature of these mechanisms. BM25 provides the anchor of exactness; dense retrieval provides the breadth of understanding.

Fusion: Reciprocal Rank Fusion (RRF)

You cannot simply add the raw scores from BM25 and a dense model together. BM25 scores are unbounded and depend on term frequency and document length. Dense similarity scores are typically bounded between -1 and 1 (or 0 and 1). Direct addition requires complex normalization and weighting that varies by dataset.

Instead, hybrid search uses Reciprocal Rank Fusion (RRF). RRF is a rank-based aggregation method that treats the output of each retriever as an independent list of ranked documents. It does not require the scores to be comparable, only the relative order. This approach is also applicable when fusing results from sparse models like splade.

The RRF formula is:

$$\text{RRF}(d) = \sum_{r \in R} \frac{1}{k + r(d)}$$

Where:

• $d$ is a document.
• $R$ is the set of rankings (e.g., BM25 ranking, Dense ranking).
• $r(d)$ is the rank of document $d$ in ranking $R$ (1-based index).
• $k$ is a constant, typically set to 60, which controls the decay of the score for lower-ranked items.

Worked Scenario

Consider a query: "how to fix java memory leak".

Retrieval Step 1: BM25 Index

1. doc_A: "Java Heap Space Exception Guide" (Rank: 1)
2. doc_B: "Python Memory Management" (Rank: 50)
3. doc_C: "JavaScript Garbage Collection" (Rank: 100)

Retrieval Step 2: Dense Index

1. doc_C: "JavaScript Garbage Collection" (Rank: 5) — Model confuses Java/JS
2. doc_A: "Java Heap Space Exception Guide" (Rank: 2)
3. doc_D: "Understanding Memory Leaks in JVM" (Rank: 1)

Fusion Step: Calculate RRF (k=60)

We calculate the RRF score for each unique document.

• For doc_A:

  • BM25 Rank: 1 $\rightarrow \frac{1}{60+1} = 0.01639$
  • Dense Rank: 2 $\rightarrow \frac{1}{60+2} = 0.01612$
  • Total RRF: $0.01639 + 0.01612 = 0.03251$

• For doc_C:

  • BM25 Rank: 100 $\rightarrow \frac{1}{60+100} = 0.00625$
  • Dense Rank: 5 $\rightarrow \frac{1}{60+5} = 0.01538$
  • Total RRF: $0.00625 + 0.01538 = 0.02163$

• For doc_D:

  • BM25 Rank: Not in top N (assume >1000) $\rightarrow \approx 0$
  • Dense Rank: 1 $\rightarrow \frac{1}{60+1} = 0.01639$
  • Total RRF: $0.01639$

Result: doc_A is ranked highest because it appeared near the top of both lists. doc_C is boosted by its high dense rank but penalized by its low BM25 rank. doc_D is retrieved solely by semantic understanding but ranks lower than doc_A because it lacked lexical confirmation.

This mechanism ensures that documents satisfying both lexical and semantic criteria are prioritized, while documents satisfying only one criterion are still considered but ranked lower.

Implementation Architecture

Implementing hybrid search requires managing two separate indexing structures and a fusion layer.

1. Data Ingestion

When ingesting documents, you must index them twice:

• Inverted Index: For BM25. Store the raw text or tokenized text. Ensure you handle stop words and stemming consistently.
• Vector Index: For dense retrieval. Generate embeddings using a model (e.g., sentence-transformers/all-MiniLM-L6-v2). Store the vector and the document ID.

2. Query Processing

At query time:

1. Tokenize: Prepare the query for BM25.
2. Embed: Pass the query through the same embedding model used for indexing.
3. Parallel Retrieval:
   • Execute a BM25 query against the inverted index.
   • Execute a vector similarity search (e.g., HNSW, IVF-PQ) against the vector index.
   • Retrieve the top $N$ results from each (e.g., $N=100$).

3. Fusion and Sorting

1. Collect Candidates: Merge the two result sets.
2. Assign Ranks: For each unique document ID, determine its rank in the BM25 list and the Dense list. If a document is not present in a list, its rank is effectively infinite (contributing 0 to RRF).
3. Calculate RRF: Apply the RRF formula.
4. Sort: Sort the merged list by descending RRF score.
5. Return Top K: Return the top $K$ documents (e.g., $K=10$).

Alternative: SPLADE

While RRF is the standard for combining separate BM25 and Dense indexes, an emerging alternative is SPLADE (Sparse Lexical and Expansion Model). SPLADE generates a sparse tensor representation that acts like BM25 but is learned by a neural network. It allows you to use a single index structure (sparse vector) that captures both lexical and semantic expansion. However, for most existing systems using Elasticsearch or OpenSearch, RRF on separate indexes is the more practical and widely supported approach.

Tradeoffs and Considerations

• Latency: Hybrid search doubles the retrieval latency compared to a single-index search. You must retrieve from two sources and compute fusion. Optimizations include reducing the candidate set size ($N$) before fusion or using approximate nearest neighbor (ANN) indexes that are highly optimized.
• Index Size: You maintain two indexes. The vector index can grow significantly in size depending on the embedding dimension and the number of vectors.
• Tuning: The parameter $k$ in RRF controls how aggressively lower-ranked items are penalized. While 60 is a common default, you may need to tune this based on your specific corpus and the relative importance of lexical vs. semantic recall. Models like splade offer a different tradeoff profile by unifying lexical and semantic signals into a single sparse output.

Hybrid search is the current state-of-the-art for general-purpose retrieval in RAG systems. It does not eliminate the need for good chunking strategies or metadata filtering, but it significantly improves the baseline relevance by ensuring that neither exact matches nor semantic nuances are lost in translation.

Conclusion

Hybrid search resolves the fundamental trade-off between lexical precision and semantic recall by fusing their independent scoring signals. By leveraging Reciprocal Rank Fusion, engineers can combine the exactness of BM25 with the breadth of dense retrieval without the complexities of score normalization. As RAG systems mature, hybrid retrieval remains the baseline for ensuring accurate information retrieval, with sparse models like splade offering viable alternatives for unified indexing.