Testing and Debugging Non-Deterministic Agents

This is Part 7 of the Building AI Agents series.

Agent testing is fundamentally different from testing traditional software. In standard engineering, we test for determinism: given input A, we expect output B. In AI agents, the core component—the Large Language Model (LLM)—is inherently stochastic. While temperature=0 enables greedy decoding, making token selection deterministic in isolation, non-determinism often persists due to external factors like network jitter, upstream model updates, or subtle context window shifts. As AI agents become more autonomous, ensuring their reliability is critical for production deployment. Treating an agent’s non-determinism as a bug to be eliminated is a category error. Instead, we must treat it as a system property and build testing infrastructure that validates correctness within a probability distribution.

This guide details how to stabilize AI agent workflows using three mechanisms: trajectory replay, strategic mocking, and semantic assertion strategies.

The Nature of the Noise

Before debugging, we must understand the source of variance. An LLM does not execute code; it predicts the next token based on a probability distribution derived from its training data and current context.

When you send a prompt to an LLM, the model calculates logits (raw prediction scores) for every token in its vocabulary. Sampling methods (like top-p or temperature scaling) introduce randomness into which token is selected. This means:

1. Exact output equality is fragile: Two runs with identical prompts may yield slightly different phrasing, order of operations, or JSON formatting.
2. Tool calling is volatile: The model may call different tools or in different orders if the decision boundary between two options is probabilistic.

Traditional unit tests that assert assert result == expected_string will fail frequently, creating "flaky tests" that erode trust in the CI/CD pipeline. To fix this, we must change what we test: we test the trajectory (the path taken) and the semantic outcome (the result’s meaning), not the exact byte sequence.

Trajectory Replay: The Regression Safety Net

The most effective way to test an agent is to record its execution history and replay it against new versions of the codebase. This is known as trajectory replay.

A trajectory consists of the sequence of steps the agent took: user input, model response, tool calls, tool outputs, and final answer. By freezing this sequence, we create a deterministic test case that isolates the agent’s logic from external noise.

How It Works

Trajectory replay works by capturing the full state of an agent's execution during a successful run. This includes the specific prompt sent to the LLM, the raw JSON response returned by the model, and the structured tool calls generated. These states are stored as fixtures (e.g., JSON files) within the test suite. During subsequent tests, these recorded inputs are injected into the agent’s components or mocked LLMs. This ensures that the agent behaves identically when re-run, effectively transforming non-deterministic behavior into a deterministic regression test. By comparing the replayed output against the expected trajectory, developers can catch logic drifts caused by code changes or prompt updates.

Example: Recording a Trajectory

Consider an agent that fetches weather data and summarizes it. A trajectory might look like this:

{
  "user_input": "What's the weather in London?",
  "steps": [
    {
      "type": "llm_call",
      "prompt": "You are a helpful assistant...",
      "response": {
        "content": null,
        "tool_calls": [
          {
            "id": "call_123",
            "name": "get_weather",
            "arguments": "{\"location\": \"London\"}"
          }
        ]
      }
    },
    {
      "type": "tool_call",
      "tool_name": "get_weather",
      "arguments": {"location": "London"},
      "result": {"temp": 15, "unit": "C", "condition": "Cloudy"}
    },
    {
      "type": "llm_call",
      "prompt": "Based on the weather data...",
      "response": {
        "content": "It is cloudy and 15 degrees Celsius in London."
      }
    }
  ]
}

Implementing Replay Tests

In your test framework, you load this fixture and inject it into the agent’s execution loop. Instead of calling the real LLM, you use a mock that returns the recorded response. This ensures that if you refactor the agent’s code, you can verify that it still makes the same decisions given the same context.

def test_weather_agent_trajectory():
    # Load recorded trajectory
    trajectory = load_fixture("weather_london.json")
    
    # Mock the LLM to return recorded responses
    mock_llm = MockLLM(trajectory.steps)
    
    # Create agent with mocked LLM
    agent = WeatherAgent(llm=mock_llm)
    
    # Run agent with user input
    response = agent.run("What's the weather in London?")
    
    # Assert the final output matches the recorded final output
    assert response.content == "It is cloudy and 15 degrees Celsius in London."

This approach transforms non-deterministic behavior into a deterministic regression test. If the agent starts making different tool calls after a code change, the test fails, alerting you to a logic drift.

Strategic Mocking of External Dependencies

Non-determinism often arises not from the LLM, but from external dependencies: APIs that return random data, databases with concurrent writes, or network latency. To isolate the agent’s reasoning, you must mock these dependencies. Effective mocking ensures that the agent’s logic is tested in isolation, free from the instability of third-party services.

Why Mock External Services?

1. Isolation: Test the agent’s logic without depending on the stability of third-party APIs.
2. Speed: Mocked responses are instant, unlike real API calls.
3. Reproducibility: Ensure tests fail only when the agent’s logic changes, not when an external API changes its schema.

Mocking Tool Calls

In frameworks like LangChain or LlamaIndex, tools are often represented as functions or classes. You can replace these with stubs that return predefined values.

from unittest.mock import patch, MagicMock
 
def test_agent_with_mocked_tool():
    # Define expected behavior
    mock_weather_result = {"temp": 20, "condition": "Sunny"}
    
    # Patch the actual tool function
    with patch('my_agent.tools.get_weather') as mock_get_weather:
        mock_get_weather.return_value = mock_weather_result
        
        # Run agent
        agent = MyAgent()
        response = agent.run("Check weather in NY")
        
        # Verify the tool was called with correct arguments
        mock_get_weather.assert_called_once_with("NY")
        
        # Verify the agent used the mocked data correctly
        assert "20 degrees" in response.content

By mocking tools, you ensure that the agent’s decision-making process is tested independently of the tool’s implementation. This is crucial for debugging: if the agent fails to interpret the weather data correctly, you know the issue is in the LLM prompt or the agent’s chain, not in the weather API.

Handling Flakiness with Semantic Assertions

Even with trajectory replay and mocking, you may encounter cases where the LLM’s output varies semantically but remains correct. For example, the agent might say "The temperature is 20°C" or "It’s 20 degrees Celsius." Both are correct, but string equality tests will fail. To handle this, use semantic assertions that evaluate the meaning of the output rather than its exact form. This is particularly important when combining trajectory replay with flexible output validation.

Strategy 1: LLM-as-a-Judge

Use a separate LLM (preferably a more stable or larger model) to evaluate the agent’s output against a set of criteria. This is known as "LLM-as-a-judge".

def test_semantic_correctness(response):
    prompt = """
    Evaluate if the following response correctly answers the user's question.
    User: What's the weather in London?
    Response: {response}
    
    Return 'PASS' if the response is accurate and relevant, otherwise 'FAIL'.
    """
    evaluator_llm = OpenAIModel(model="gpt-4o")
    result = evaluator_llm.run(prompt.format(response=response))
    
    assert "PASS" in result.content

This approach is robust because it allows for natural language variations. However, it introduces its own latency and cost. Use it for critical validation paths, not every test.

Strategy 2: Embedding Similarity

For tasks where the output is a vector of information (e.g., summaries, code snippets), compare the embedding of the actual output against the expected output. If the cosine similarity is above a threshold (e.g., 0.9), consider the test passed.

import numpy as np
from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer('all-MiniLM-L6-v2')
 
def test_embedding_similarity(actual, expected):
    actual_emb = model.encode(actual)
    expected_emb = model.encode(expected)
    
    similarity = np.dot(actual_emb, expected_emb) / (np.linalg.norm(actual_emb) * np.linalg.norm(expected_emb))
    
    assert similarity > 0.9, f"Outputs are too dissimilar: {similarity}"

This method is fast and effective for comparing semantic content. It is particularly useful for testing retrieval-augmented generation (RAG) systems, where the exact wording of the retrieved context may vary.

Debugging Strategies for Non-Determinism

When tests fail, debugging non-deterministic agents requires a systematic approach. By identifying the root cause of variance, you can reduce the occurrence of flaky tests and improve overall system stability.

1. Trace Analysis: Use tracing tools (like LangSmith or Arize Phoenix) to visualize the agent’s execution. Look for steps where the model’s output diverges from the expected trajectory. Identify if the divergence is due to a tool call, a prompt change, or a context window overflow.
2. Prompt Sensitivity Testing: If an agent is flaky, try varying the prompt slightly to see if the output becomes more stable. Sometimes, adding explicit instructions or examples (few-shot prompting) reduces variance.
3. Temperature Adjustment: If the agent’s task is factual and requires precision, reduce the temperature to 0 or 0.1. For creative tasks, higher temperatures are acceptable, but you must rely more on semantic assertions.
4. Context Window Management: Large context windows can introduce noise. Truncate irrelevant parts of the context to improve focus. Test with different context sizes to identify sensitivity.

Conclusion

Testing non-deterministic agents requires a shift in mindset. We cannot expect byte-for-byte reproducibility. Instead, we build robustness through:

• Trajectory replay to catch logic drift.
• Strategic mocking to isolate agent logic from external noise.
• Semantic assertions to validate correctness despite linguistic variation.

By implementing these strategies, you transform flaky, unreliable tests into a stable foundation for AI agent development. This allows you to iterate confidently, knowing that your agents behave correctly within the bounds of their probabilistic nature.