Task-Specific Metrics for RAG, Agents, and Extraction

Standard evaluation metrics like BLEU or ROUGE measure token overlap between a generated response and a reference text. This works for machine translation where the target structure is fixed. It fails for Retrieval-Augmented Generation (RAG), Agents, and Information Extraction (IE) because these systems are generative and stateful. A model can produce a factually correct answer with low token overlap, or a syntactically perfect answer that is completely hallucinated. To evaluate these systems, we must move from surface similarity to mechanism validation. We need metrics that verify the logical chain of events: retrieval, reasoning, and output formatting.

RAG: Faithfulness and Groundedness

In a RAG system, the LLM receives retrieved context snippets and a user query to generate an answer. The failure modes here are distinct: the model might ignore the context, or it might contradict the context. We measure these using two orthogonal metrics: Groundedness and Faithfulness.

Faithfulness measures whether the answer can be inferred from the provided context. It checks for hallucinations relative to the source material. If the context says "The sky is blue" and the model says "The sky is green," the faithfulness score is zero. This is typically evaluated by breaking the answer into atomic statements and checking if each statement is entailed by the context.

Groundedness measures whether the answer directly addresses the user's query using the provided context. It ensures the model didn't just retrieve irrelevant information or hallucinate details not present in the prompt. A high groundedness score means the answer is a logical conclusion derived from the context, not external knowledge.

Consider a query: "What is the capital of France?" with context: "Paris is a city in France known for the Eiffel Tower."

• Answer: "Paris is the capital of France."
• Faithfulness: Low. The context does not state that Paris is the capital. The model used external knowledge.
• Groundedness: High. The answer is consistent with the context (Paris is in France) but fails the strict grounding test if the system requires strict adherence.

Conversely, if the context contains the capital and the model uses it correctly, both scores are high. The mechanism here relies on pairwise entailment models. We split the answer into claims $C_1, C_2, ... C_n$ and check if $Context \implies C_i$ for all $i$. If any $C_i$ cannot be proven by the context, faithfulness drops.

Agents: Trajectory Metrics

Agents differ from RAG because they execute actions, observe results, and update their internal state over multiple steps. Evaluating an agent requires looking at the entire trajectory, not just the final output. A final answer might be correct, but if the agent took a dangerous or inefficient path, the system is flawed.

We use trajectory metrics to evaluate the sequence of actions. This involves two layers:

1. Step-Level Correctness: Does the action taken at step $t$ logically follow from the state at $t-1$ and the available tools? For example, if an agent needs to calculate $2+2$, it should call a calculator tool, not guess.
2. Outcome Validity: Did the sequence of actions achieve the goal?

Consider an agent tasked with "Find the weather in London and tell me if I need an umbrella."

• Trajectory:
  1. Action: Call get_weather(location="London")
  2. Observation: {"temp": 10, "rain": true}
  3. Action: Call analyze_rain_need(temp=10, rain=true)
  4. Final Answer: "Yes, bring an umbrella."

If the agent skipped step 2 and guessed "It's raining," the outcome might be correct, but the trajectory is invalid. The step-level metric would flag this as a failure because the model did not use the available tool. The mechanism for this evaluation often involves simulating the trajectory in a sandbox or using a separate verifier LLM to judge the logical coherence of each step.

Information Extraction: Semantic Accuracy

Information Extraction (IE) involves pulling structured data (JSON, SQL, CSV) from unstructured text. Here, token-based metrics are useless because the structure is rigid. A missing comma or a wrong data type breaks the entire pipeline. We need extraction accuracy, which measures semantic equivalence and schema adherence.

There are two components:

1. Schema Compliance: Does the output match the expected JSON structure? Are all required keys present?
2. Value Correctness: Are the extracted values semantically equivalent to the source?

Consider extracting a date from "Meeting scheduled for tomorrow, Nov 5th."

• Expected Output: {"date": "2023-11-05"}
• Model Output: {"date": "Nov 5"}

A string comparison fails here. A semantic comparison must recognize that "Nov 5" and "2023-11-05" refer to the same temporal point. This requires a normalization step. For numerical values, we check if the extracted number falls within a tolerance range of the true value. For categorical values, we check if the extracted category matches the source intent.

The mechanism for IE evaluation often involves parsing the output and running unit tests against the source text. If the extraction system fails to parse the JSON, the accuracy is zero. If it parses but extracts the wrong entity, the accuracy is zero. This binary nature makes IE evaluation stricter than open-ended generation.

Synthesis: Choosing the Right Metric

The choice of metric depends on the system's failure mode.

• If your RAG system returns irrelevant answers, optimize for Groundedness.
• If your RAG system hallucinates facts, optimize for Faithfulness.
• If your Agent takes inefficient paths, optimize for Trajectory Metrics.
• If your IE system breaks downstream pipelines, optimize for Extraction Accuracy.

These metrics are computationally more expensive than BLEU or ROUGE because they often require multiple LLM calls or specialized verifiers. However, they provide the only actionable signal for improving complex AI systems. Surface similarity is not enough; we must measure the logic.

Common Pitfalls in Metric Implementation

Implementing these task metrics introduces several operational challenges that teams often overlook:

1. Cost of Verification: Using LLMs as judges for faithfulness or trajectory steps adds significant latency and cost. A single query might trigger 5-10 verification calls. Mitigate this by using smaller, distilled models for initial screening before invoking larger, more accurate verifiers.
2. Judge Bias: LLM-based evaluators can inherit biases from their training data. For example, a judge might unfairly penalize concise answers if it prefers verbose explanations. Always calibrate your judge against a gold-standard dataset to ensure consistency.
3. Over-Optimization: Optimizing strictly for faithfulness can sometimes reduce helpfulness if the context is incomplete. You must balance strict grounding with the user's need for a complete answer, potentially allowing the model to cite its confidence when relying on internal knowledge.

Conclusion

Evaluating modern AI systems requires a shift from statistical overlap to logical verification. By implementing faithfulness and groundedness for RAG, trajectory metrics for agents, and semantic accuracy for extraction, engineers can identify specific failure modes rather than relying on noisy, surface-level scores. This precision allows for targeted optimization, ensuring that complex systems behave reliably in production environments.