Evaluating Agentic RAG: Trajectory, Ragas & Metrics

Evaluating Agentic RAG Systems

Traditional Retrieval-Augmented Generation (RAG) evaluation assumes a linear pipeline: retrieve context, generate answer. Agentic RAG breaks this linearity. An agent may retrieve, reason, call an API, retrieve again, and then synthesize. Evaluating this system using only end-to-end answer correctness is insufficient because it masks why the system failed. Did it fail because of poor attribution of the error to the retrieval layer or the planning logic?

This is Part 6 of the Agentic RAG Patterns series. To evaluate agentic RAG effectively, ML engineers must shift from evaluating static outputs to evaluating dynamic trajectories. This shift in rag evaluation from static outputs to dynamic trajectories is critical for agentic systems. This requires a layered approach: stage-specific metrics for retrieval and generation components, and trajectory-level validation for the agent’s decision-making loop.

The Failure of Linear Metrics

In a standard RAG system, metrics like Context Recall and Faithfulness (from the Ragas framework) are sufficient. You compare the generated answer against the ground truth and the retrieved context. However, in an agentic system, the context is not static. The agent actively constructs the context through tool use.

Consider a scenario where an agent needs to find the current stock price of a company. A linear RAG system might search a static knowledge base, find outdated info, and hallucinate. An agentic system will use a search_api tool. If the tool call fails, the agent might retry or switch strategies. Standard RAG metrics cannot capture the validity of the tool call itself. They only see the final text. Therefore, we need to decompose the evaluation into two distinct mechanisms: Stage Evaluation and Trajectory Evaluation.

Stage-Specific Metrics: Isolating Components

Before evaluating the agent’s brain, we must ensure its sensory organs (retrieval) and voice (generation) are functioning correctly. This is where stage metrics from frameworks like Ragas remain valuable, but they must be applied to specific sub-tasks within the agent’s loop.

Retrieval Stage: Precision and Recall at Each Step

For every retrieval action the agent takes, we can calculate standard RAG metrics. If the agent performs three retrieval steps, we evaluate each independently.

# Conceptual example of evaluating a single retrieval step
def evaluate_retrieval_step(retrieved_docs, ground_truth_context):
    # Calculate Context Precision: Did the agent pick the right docs?
    # Note: calculate_precision is a placeholder for the specific 'Context Precision' metric implementation from the evaluation framework being used.
    precision = calculate_precision(retrieved_docs, ground_truth_context)
    # Calculate Context Recall: Did it miss any necessary docs?
    recall = calculate_recall(retrieved_docs, ground_truth_context)
    return {"precision": precision, "recall": recall}

These stage metrics isolate the performance of individual components, preventing the masking of errors in the final output. This isolates the retriever’s performance. If retrieval scores are low, the problem is in the embedding model or chunking strategy, not the agent’s planning. If retrieval scores are high but the final answer is wrong, the failure lies in the agent’s reasoning or synthesis stage.

Generation Stage: Faithfulness and Answer Relevance

Once the agent has gathered all necessary information, it generates a response. Here, we apply Ragas’ faithfulness metric to ensure the agent did not hallucinate facts not present in the accumulated context. We also use answer_relevancy to ensure the answer directly addresses the user’s query, which is critical when agents may get sidetracked by irrelevant tool outputs.

Trajectory Evaluation: Validating the Logic

The core challenge in agentic RAG is that the path to the answer matters. Two agents might produce the same correct answer, but one might have used a valid, efficient sequence of tools, while the other made three unnecessary API calls before guessing correctly. Trajectory evaluation assesses the process.

Automated trajectory eval allows us to score the logical flow without manual inspection of every step.

Parsing the Execution Trace

Every agentic run generates a trace—a JSON structure detailing the sequence of thoughts, actions, and observations. To evaluate this, we must parse the trace to extract the logical flow.

{
  "query": "What is the capital of France?",
  "trajectory": [
    {"step": 1, "action": "search", "params": {"query": "France capital"}},
    {"step": 2, "observation": "Paris"},
    {"step": 3, "action": "finalize", "result": "Paris"}
  ]
}

Trajectory evaluation involves checking these steps against a set of rules or a reference trajectory. Common checks include:

1. Tool Correctness: Did the agent use the correct tool for the intent?
2. Parameter Validity: Were the parameters passed to the tool valid?
3. Loop Termination: Did the agent stop when it had enough information, or did it enter an infinite loop?

Automated Trajectory Scoring

We can automate this by using a secondary LLM as a judge, or by using rule-based parsers for structured traces. The judge evaluates the sequence: "Given the query and the intermediate steps, was the final action justified?"

This is distinct from answer evaluation. An agent might retrieve the wrong document but still guess the correct answer. Trajectory evaluation flags this as a failure because the reasoning was unsound, even if the output was lucky. This is crucial for safety-critical applications where consistency matters more than occasional correctness.

Integrating Ragas with Trajectory Analysis

Ragas does not natively support trajectory evaluation. However, we can integrate it by treating each retrieval step as a separate RAG instance. For the final synthesis, we treat the accumulated context as the "retrieved documents."

By integrating Ragas with our custom trace parsers, we leverage the robustness of the ragas framework for component-level checks.

The workflow looks like this:

1. Run the Agent: Capture the full trace.
2. Extract Sub-Queries: Identify each time the agent performed a retrieval.
3. Evaluate Sub-Queries: Run Ragas metrics on each retrieval step.
4. Evaluate Final Synthesis: Run Ragas metrics on the final answer against the combined context.
5. Evaluate Trajectory: Use a trace parser or LLM-judge to score the logical flow.

This hybrid approach provides a comprehensive framework for rag evaluation, ensuring both component accuracy and logical soundness. This hybrid approach gives us a comprehensive view. We know if the retriever is good, if the generator is faithful, and if the agent is thinking logically.

Conclusion

Evaluating agentic RAG systems is not about finding a single metric that captures everything. It is about decomposing the system into its constituent parts and evaluating each according to its mechanism. Stage metrics (Ragas) ensure the components work correctly. Trajectory evaluation ensures the agent’s logic is sound. By combining these, ML engineers can move beyond black-box testing and build agents that are not just accurate, but reliable and explainable.