Building an Eval Harness with promptfoo

Evaluating Large Language Models (LLMs) is not a product feature; it is a deployment risk. Unlike traditional software where unit tests verify logic against deterministic outputs, LLM outputs are probabilistic. A prompt change that improves helpfulness might inadvertently reduce safety, or a temperature tweak might break a strict formatting requirement. To manage this variance, engineering teams need an eval harness—a system that treats prompt changes like code changes, subjecting them to rigorous, automated testing before they reach production.

This guide details how to build that harness using promptfoo, an open-source evaluation framework. We will move beyond subjective "does this look good?" checks to a mechanism-driven approach involving structured assertions, continuous integration (CI) gates, and statistical prompt comparison.

The Evaluation Model: Separation of Concerns

At its core, promptfoo implements a decoupled evaluation architecture. It separates three distinct components:

1. The Provider: The LLM API (e.g., OpenAI, Anthropic, local models via Ollama) that generates the response.
2. The Test Suite: A dataset of input prompts and expected outcomes (ground truth).
3. The Assertion Engine: The logic that compares the generated response against the expected outcome.

This separation allows you to swap models without rewriting tests, and swap assertion logic without changing the model provider. The harness does not judge the quality itself; it executes the judgment criteria defined by the engineer.

Defining Assertions Mechanistically

The strength of promptfoo lies in its assertion types. These are not just string matches; they are specialized validators that operate at different levels of abstraction.

Consider a customer support bot that must always adhere to a refund policy. A naive test might check if the word "refund" appears in the response. However, a more robust harness uses layered assertions.

In promptfoo, these are defined in a configuration file (typically promptfooconfig.yaml). Here is how you define a test case with multiple assertion layers:

prompts:
  - file://prompts/support_bot.md
 
providers:
  - openai:gpt-4o
 
tests:
  - vars:
      user_query: "I want a refund for my broken toaster."
      policy: "Refunds are only allowed within 30 days of purchase."
    
    # Assertion 1: Deterministic check
    assert:
      - type: contains
        value: "refund"
      
      # Assertion 2: LLM-as-a-judge
      - type: llm-rubric
        value: "The model should reference the 30-day policy limit."
        
      # Assertion 3: Custom JavaScript logic
      - type: javascript
        value: "return output.includes('30') || output.includes('month');"

Mechanism Breakdown

1. contains: A deterministic string match. This is fast and cheap. It verifies that specific keywords are present. If the policy requires the phrase "30 days," this assertion fails immediately if the model hallucinates "60 days."
2. llm-rubric: This uses a separate, often cheaper or more capable LLM instance to evaluate the output based on a natural language rubric. The mechanism here is prompt injection: promptfoo sends the original prompt, the user query, and the generated response to the judge model with instructions like "Evaluate if the response adheres to the following rubric: [rubric]." The judge returns a score or pass/fail. This captures semantic correctness that regex cannot.
3. javascript: This injects a Node.js function that receives the model's output as a variable. This allows for complex logic, such as parsing JSON responses, checking data types, or performing arithmetic. For example, if the model returns a structured JSON object containing a confidence score, you can assert that the score is above 0.8.

Advanced JavaScript Assertion Example

For more complex structures, you can parse JSON directly. Suppose your model returns a structured error code. You can validate the schema using JavaScript:

// Assertion value in promptfooconfig.yaml
try {
  const json = JSON.parse(output);
  return json.status === 'error' && typeof json.code === 'number' && json.code > 0;
} catch (e) {
  // Fail if response is not valid JSON
  return false;
}

If an assertion fails, promptfoo logs the specific failure reason and the actual output, allowing engineers to debug whether the issue lies in the prompt, the model, or the test definition itself.

The CI Integration Loop

An eval harness is useless if it runs only on your laptop. The critical engineering step is integrating promptfoo into your CI/CD pipeline. This turns evaluation into a gatekeeper.

Data Flow in CI

1. Commit: Developer modifies a prompt in prompts/support_bot.md.
2. Trigger: GitHub Actions (or GitLab CI) detects the change.
3. Execution: The CI runner installs promptfoo and executes the eval suite against the modified prompt.
4. Comparison: promptfoo compares the new scores against a baseline (either stored in the repo or fetched from a previous run).
5. Gate: If the score drops below a threshold (e.g., 95%), the build fails.

GitHub Actions Example

Here is a concrete implementation of the CI step. This job runs the evals and fails if any assertion fails.

name: LLM Eval Harness
 
on: [push]
 
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20
 
      - name: Install promptfoo
        run: npm install -g promptfoo
 
      - name: Run Evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          # Run evals and generate a report
          promptfoo eval --config promptfooconfig.yaml --output eval-results.json
          
          # Fail the build if any test fails
          # promptfoo exits with non-zero status on assertion failure
          promptfoo eval --config promptfooconfig.yaml --no-progress-bar

Why This Matters

Without this loop, prompt changes are blind. A developer might optimize a prompt for brevity, inadvertently breaking a downstream parser. The CI gate catches this regression immediately. The promptfoo CLI returns an exit code of 1 if any assertion fails, which is the standard mechanism for CI failure detection.

Prompt Comparison and Diffing

One of the most challenging aspects of LLM development is knowing why a score dropped. Was it the model update? The prompt change? The test case?

promptfoo addresses this through its prompt comparison feature. You can define multiple providers or prompt variations in a single config and run them against the same test suite.

Side-by-Side Comparison

providers:
  - openai:gpt-4o
  - openai:gpt-3.5-turbo
 
prompts:
  - file://prompts/v1/support_bot.md
  - file://prompts/v2/support_bot.md

When you run promptfoo eval with this config, it generates a diff report. This report shows:

1. Aggregate Scores: Average pass rate per provider/prompt combo.
2. Individual Test Diffs: For each test case, it shows which provider/prompt passed or failed.
3. Failure Analysis: It highlights exactly where the new prompt diverged from the old one.

Statistical Significance in Small N

A common pitfall is assuming that a 1% improvement in a 10-test suite is significant. promptfoo encourages large test suites (hundreds or thousands of examples) to achieve statistical relevance. When comparing prompts, the harness aggregates results across the entire suite, smoothing out noise from individual edge cases.

For example, if v1 has a 90% pass rate and v2 has a 92% pass rate on a 500-example suite, that two-point difference could easily fall within normal run-to-run variance. Without a statistically rigorous comparison, the improvement may not be meaningful, and the new prompt should not be deployed based on this data alone. This prevents engineers from chasing noise.

Pitfalls

While promptfoo provides a strong foundation, several pitfalls can undermine your evaluation efforts.

1. Over-reliance on LLM-as-a-Judge for Simple Tasks: Using an expensive LLM to check for simple string matches (like "contains 'yes'") is inefficient and introduces unnecessary latency and cost. Use deterministic assertions (contains, regex) for factual checks and reserve llm-rubric for nuanced, semantic evaluations.
2. Ignoring Cost Implications: Running large test suites against expensive models (like GPT-4) can incur significant costs. Always cache results when possible, use cheaper models for initial iterations, and monitor your API usage. Consider using local models for rapid iteration if hardware permits.
3. Static Test Suites: LLM behavior drifts over time. If your test suite is not regularly updated to reflect new edge cases or model updates, your eval harness will become stale and less effective at catching regressions.

Takeaways

To build an effective eval harness, keep these engineering principles in mind:

• Separation of Concerns: Keep prompts, tests, and evaluation logic distinct. This allows for independent iteration and easier maintenance.
• CI Integration: Automation is key. Integrate promptfoo into your CI/CD pipeline to ensure every prompt change is evaluated before deployment.
• Statistical Significance: Use large, diverse test suites to ensure that your evaluation results are meaningful and not the result of random chance.

FAQ

How do I handle non-deterministic outputs? LLM outputs are inherently probabilistic. To handle this, run your tests multiple times with different temperatures or seeds and aggregate the results. promptfoo supports running tests multiple times to calculate aggregate pass rates, giving you a more accurate picture of reliability.

Can I use promptfoo with local models? Yes. promptfoo supports various providers, including local models via Ollama, LM Studio, or any OpenAI-compatible API endpoint. You can configure local providers in your promptfooconfig.yaml just like remote APIs, allowing you to test locally before deploying to production.

What happens if an assertion fails? When an assertion fails, promptfoo records the failure in the output report (JSON, CSV, or HTML). In a CI environment, you can configure the tool to fail the build, preventing the integration of broken prompts. You can also review the specific output that caused the failure to debug the issue.

Conclusion

Building an eval harness with promptfoo shifts LLM development from an art to an engineering discipline. By structuring evaluations as code, defining assertions as deterministic and semantic validators, and integrating these checks into CI, you create a feedback loop that catches regressions before they reach users.

The key takeaway is not just the tool, but the mechanism: separation of concerns. Keep your prompts, tests, and evaluation logic distinct. Let the harness handle the execution, and let your engineering judgment handle the design. This structure ensures that as your LLM applications grow in complexity, your ability to test them scales with them.