Measuring Retrieval Quality: Recall@k, nDCG, and MRR

In traditional supervised learning, we optimize for accuracy, precision, or F1-score. These metrics assume a closed world: a fixed set of classes and a clear distinction between positive and negative examples. Retrieval systems operate in an open world. When a user queries "Jaguar," the system must decide whether to return a car manual, a wildlife documentary, or a software framework. There is no single "correct" answer in the classical sense; there is only a graded set of relevant documents among a universe of non-relevant ones.

Evaluating such systems requires metrics that disentangle two distinct problems: recall (did we find the relevant items?) and ranking (did we place them in the order the user prefers?). This article examines three foundational metrics—Recall@k, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (nDCG)—to explain how they measure different aspects of retrieval quality.

The Failure of Binary Accuracy

Consider a search index containing 10,000 documents. A user queries for "Python." Suppose only 50 documents are truly relevant to the user’s intent (programming tutorials), while 9,950 are irrelevant (zoology facts).

If a retrieval system returns 10 documents, and 5 of them are relevant, a naive accuracy metric might calculate $5/10 = 50\%$. However, this ignores the fact that the system failed to retrieve 45 relevant documents from the index. Conversely, if the system returned all 10,000 documents, accuracy would be $50/10,000 = 0.5\%$, which is misleadingly low despite perfect recall.

Retrieval metrics must account for the pool of available items and the position of relevant items within the returned list.

Recall@k: Measuring Breadth

Recall@k measures the fraction of relevant items that appear within the top $k$ results. It is a binary metric: a document is either relevant or not. It answers the question: "Did we find everything that matters?"

Mechanism

Let $R$ be the set of all relevant documents in the corpus for a given query. Let $A_k$ be the set of the top $k$ documents retrieved by the system. Recall@k is defined as:

$\text{Recall@k} = \frac{|R \cap A_k|}{|R|}$

The denominator $|R|$ is the ground truth count of relevant items. The numerator counts how many of those landed in the top $k$.

Worked Scenario

Imagine a legal research assistant. A lawyer queries for "precedent regarding contract voidability." In the database, there are exactly 10 relevant case law documents ($|R| = 10$).

System A returns the top 5 results ($k=5$). Among these 5, 4 are relevant cases. $\text{Recall@5} = \frac{4}{10} = 0.4$

System B returns the top 20 results ($k=20$). Among these 20, 9 are relevant cases. $\text{Recall@20} = \frac{9}{10} = 0.9$

System B has higher recall because it captured more of the relevant universe. However, Recall@k does not tell us where the relevant items were. If System A’s 4 relevant items were at positions 1, 2, 3, and 4, it is far more useful than if they were at positions 16, 17, 18, and 19. Recall@k treats all positions within $k$ equally.

Use Case: Recall@k is critical in domains where missing an item is catastrophic. Examples include medical diagnosis support, legal discovery, and security threat detection. If a threat actor's signature is in the corpus but ranked below $k$, the system has failed.

Mean Reciprocal Rank (MRR): Measuring Position of First Hit

MRR focuses on the position of the first relevant document. It assumes that if the user’s intent is satisfied by the first relevant result, subsequent results are less important. It answers the question: "How quickly did we find the first correct answer?"

Mechanism

For a single query, the Reciprocal Rank (RR) is the inverse of the rank position of the first relevant document. If the first relevant document is at position 1, RR = 1/1 = 1. If it is at position 3, RR = 1/3. If no relevant document is found, RR = 0.

MRR is the average of RR across $N$ queries:

$\text{MRR} = \frac{1}{N} \sum_{i=1}^{N} \frac{1}{\text{rank}_i}$

Worked Scenario

Consider a developer looking for a specific JAR file version: lombok-1.18.30.jar.

Query 1: The correct file is at position 1. $\text{RR}_1 = 1/1 = 1.0$

Query 2: The correct file is at position 2. $\text{RR}_2 = 1/2 = 0.5$

Query 3: The correct file is at position 5. $\text{RR}_3 = 1/5 = 0.2$

$\text{MRR} = \frac{1.0 + 0.5 + 0.2}{3} = 0.567$

MRR penalizes deeper ranks heavily. Moving a relevant result from position 1 to 2 halves the score. Moving it from 2 to 3 reduces it from 0.5 to 0.33. This steep discount reflects the user behavior that most users stop searching after the first relevant result.

Use Case: MRR is ideal for navigation tasks, question-answering bots, and URL shorteners where the user expects a single, precise answer. It is less suitable for exploratory search where multiple relevant items are desired.

Normalized Discounted Cumulative Gain (nDCG): Measuring Graded Relevance

Real-world relevance is rarely binary. A document about "Python programming" is highly relevant to a query for "Python tutorials," moderately relevant to "Python snake facts," and irrelevant to "Python library imports." nDCG captures this graded relevance and penalizes lower-ranked relevant items.

Mechanism

nDCG is derived from DCG (Discounted Cumulative Gain).

1. Calculate Gain: Assign a relevance score $rel_i$ to each document at position $i$ (e.g., 0=irrelevant, 1=less relevant, 2=highly relevant).
2. Discount Position: Apply a logarithmic discount to reflect that users value top results more. The gain at position $i$ is: $\text{Gain}_i = \frac{2^{rel_i} - 1}{\log_2(i + 1)}$
3. Sum Gain: Sum these values up to position $k$ to get DCG@k.
4. Normalize: Divide DCG@k by the Ideal DCG (IDCG@k), which is the maximum possible DCG achievable if all relevant documents were perfectly ranked at the top.

$\text{nDCG@k} = \frac{\text{DCG@k}}{\text{IDCG@k}}$

Worked Scenario

Query: "Best Python frameworks for web dev." Relevance scale: 0 (Irrelevant), 1 (Relevant), 2 (Highly Relevant). $k=3$.

System A (Ideal Ranking):

• Pos 1: "Django Guide" (Rel=2)
• Pos 2: "Flask Tutorial" (Rel=2)
• Pos 3: "FastAPI Docs" (Rel=2)

$\text{DCG@3}_A = \frac{2^2-1}{\log_2(2)} + \frac{2^2-1}{\log_2(3)} + \frac{2^2-1}{\log_2(4)}$ $\text{DCG@3}_A = \frac{3}{1} + \frac{3}{1.585} + \frac{3}{2} \approx 3 + 1.89 + 1.5 = 6.39$ Since this is perfectly ranked, $\text{IDCG@3} = 6.39$. $\text{nDCG@3}_A = 1.0$

System B (Poor Ranking):

• Pos 1: "Snake Care" (Rel=0)
• Pos 2: "Django Guide" (Rel=2)
• Pos 3: "Flask Tutorial" (Rel=2)

$\text{DCG@3}_B = \frac{2^0-1}{\log_2(2)} + \frac{2^2-1}{\log_2(3)} + \frac{2^2-1}{\log_2(4)}$ $\text{DCG@3}_B = 0 + 1.89 + 1.5 = 3.39$ $\text{nDCG@3}_B = \frac{3.39}{6.39} \approx 0.53$

System B has the same relevant items as System A, but because they are pushed down, its nDCG is significantly lower. nDCG rewards systems that place more relevant items higher, not just the first one.

Use Case: nDCG is the standard for content recommendation systems (Netflix, Spotify), news feeds, and e-commerce search. In these domains, user satisfaction is a function of the cumulative relevance of the top $k$ items, not just a single hit.

Synthesizing the Metrics

Choosing the right metric depends on the quality of your relevance judgments and the user’s interaction model.

1. Recall@k is for completeness. Use it when the cost of missing a relevant item is high. It is insensitive to rank order within $k$.
2. MRR is for precision of intent. Use it when the user seeks a single, definitive answer. It is highly sensitive to the position of the first relevant item.
3. nDCG is for utility. Use it when relevance is graded and the user consumes multiple items. It balances both the quantity of relevant items and their ranking.

Engineers often refer to this metric as recall at k when discussing its sensitivity to the cutoff parameter $k$. In practice, these metrics complement each other in a real-world evaluation pipeline. Recall@k serves as a baseline check to ensure the retrieval system isn't failing to find relevant documents entirely. Once completeness is verified, optimization typically shifts toward nDCG to improve the ranking quality of those retrieved items. A system might have high Recall@k (finding all items) but low nDCG (ranking them poorly). Conversely, a system might have high MRR (hitting the top item first) but low Recall@k (missing many other relevant items). Understanding the mechanism behind each metric allows you to diagnose whether the retrieval bottleneck is in the matching phase (recall) or the scoring/ranking phase (position).

Conclusion

Retrieval evaluation is not a single number but a dimensional analysis of relevance (did we find it?) and ranking (was it in the right order?). Recall@k, MRR, and nDCG solve specific failure modes in information retrieval by targeting breadth, depth, and graded utility respectively. By selecting the metric that aligns with your user's interaction model, you can accurately measure and improve your system's performance.