Skip to content
Ashish.
All posts
Diagram illustrating the pipeline from production logs to curated golden dataset.
6 min readDevelopmentML engineers, Product engineersFeatured#llm-evaluation#golden-dataset#production-traffic#data-sampling#labeling#ml-engineering#eval-curation

Building a Golden Dataset From Production Traffic

A practical guide for ML engineers on curating high-quality golden datasets by sampling and labeling production traffic for LLM evaluation.

By Ashish KumarPart 2 of LLM Evaluation Engineering

Most LLM evaluation suites fail in production because they are built on synthetic data that doesn’t reflect the messy, ambiguous nature of real user queries. A golden dataset isn’t just a collection of questions and answers; it’s a statistically representative snapshot of what your users actually do. To build one, you must extract, sample, and label production traffic with the same rigor used for training data. This is Part 2 of the LLM Evaluation Engineering series.

Why Synthetic Data Fails

A schematic comparison diagram showing two JSON objects side-by-side : a 'Synthetic' example with simple, average phrasing and a 'Production' example with complex, messy, long-tail user queries, highlighting the distribution gap.

Synthetic datasets are generated by prompting an LLM to create variations of a base question. While scalable, this process suffers from distribution collapse. The synthetic generator tends to converge on "average" phrasing, missing the long-tail edge cases that cause production failures.

For example, a synthetic generator might produce:

{"query": "How do I reset my password?", "answer": "Go to the login page..."}

But production traffic contains:

{"query": "I'm locked out and my email changed last week, help", "answer": "Complex support flow required..."}

Evaluating against synthetic data creates a false sense of security. The model passes the test but fails in production because the evaluation distribution never included the high-entropy, low-frequency queries that define real-world robustness. Furthermore, synthetic data often lacks the contextual nuance present in human interactions, such as sarcasm, incomplete sentences, or domain-specific jargon that only emerges from actual usage patterns. This gap means that models optimized on synthetic benchmarks may appear competent in testing but struggle significantly when faced with the unpredictable complexity of live user inputs.

Ingestion and Sampling Strategy

The first step is ingesting production traffic without impacting latency. This is typically done via async logging to a data lake or warehouse (e.g., S3, BigQuery, Snowflake).

1. Log Extraction

Capture the following fields for every request:

  • user_id (hashed for privacy)
  • query (raw input)
  • response (model output)
  • latency_ms
  • timestamp
  • session_id (to group related turns)

2. Stratified Sampling

A flowchart diagram illustrating the stratified sampling process, showing production logs being divided into distinct strata (e.g., by query length or keyword) and then sampled proportionally from each group.

Random sampling often misses rare but critical failure modes. Instead, use data sampling techniques like stratified sampling based on features that correlate with model difficulty.

Mechanism: Divide your production log into strata (subgroups) based on a proxy for complexity, such as:

  • Query length (short vs. long)
  • Presence of specific keywords (e.g., "error", "bug", "cancel")
  • User tier (free vs. enterprise)

Then, sample proportionally from each stratum to ensure the golden dataset reflects the production mix. For instance, if 5% of production queries are "complaints," your golden dataset should contain at least 5% complaint samples, even if they are rare.

# Pseudocode for stratified sampling
import pandas as pd
 
# Load production logs
df = load_production_logs()
 
# Define strata
df['complexity'] = df['query'].apply(lambda x: classify_complexity(x))
 
# Sample 1000 items per complexity level
sampled_data = []
for level in df['complexity'].unique():
    subset = df[df['complexity'] == level]
    sampled = subset.sample(n=1000, random_state=42)
    sampled_data.append(sampled)
 
golden_df = pd.concat(sampled_data)

Labeling and Quality Control

A raw sample is not a golden instance. It must be labeled for correctness, safety, and helpfulness. This is the most labor-intensive step in the ML engineering lifecycle. Effective eval curation ensures that the resulting dataset is not just large, but representative and high-fidelity.

1. Human-in-the-Loop Annotation

Deploy a labeling interface where annotators review the (query, response) pair. Use a structured rubric:

  • Correctness: Is the factual content accurate?
  • Safety: Does the response violate policy (e.g., PII leakage, harmful content)?
  • Helpfulness: Does the response actually solve the user’s problem?

2. Inter-Annotator Agreement (IAA)

To ensure consistency, have multiple annotators label the same subset. Calculate Cohen’s Kappa or Fleiss’ Kappa to measure agreement. If IAA is low, the rubric is ambiguous. Refine the guidelines and retrain annotators.

3. LLM-as-a-Judge (Optional)

For large-scale curation, use a stronger LLM to pre-label samples. However, this introduces bias toward the judge model’s preferences. Always spot-check a random subset of LLM-labeled samples with humans to calibrate the judge.

Conclusion

A golden dataset from production traffic is not a static artifact. It’s a living representation of your user base. By sampling strategically and labeling rigorously, you ensure your evaluation metrics reflect real-world performance, not just synthetic perfection. This process reduces the gap between test-time accuracy and production reliability, making your LLM evaluation a true indicator of business value.

Next time you build an eval suite, ask: "Does this test cover the queries my users actually send?" If the answer is no, go back to production logs.

Related posts