Evaluating a Fine-Tune Against the Base Model

When you fine-tune a Large Language Model (LLM), you are not simply adding new knowledge; you are altering the weight distribution of a model that already possesses vast general capabilities. This process carries an inherent risk: catastrophic forgetting. As the model adjusts its weights to optimize for your specific domain data, it may degrade its ability to perform well on general tasks it previously mastered. Therefore, evaluating a fine-tune is not merely about measuring how well it does on your test data. It is a regression test. You must compare the fine-tuned model against the base model to quantify exactly what was gained and what was lost.

Part 7 of the Fine-Tuning LLMs Hands-On series.

The Mechanism of Forgetting

To understand why we need a base model comparison, we must look at the optimization landscape. As noted in recent studies on catastrophic forgetting (e.g., Kirkpatrick et al., 2017), the phenomenon occurs when new learning overwrites previously acquired knowledge. During pre-training, the model learns a broad representation of language. During fine-tuning, gradient descent updates these weights to minimize loss on a narrower distribution. If the fine-tuning dataset is small or biased, the optimization trajectory can push the model’s parameters away from regions that support general reasoning, leading to a phenomenon known as "task interference" or "catastrophic forgetting."

This is not a bug; it is a feature of how neural networks store information. Optimizing for specific syntax shifts representational focus, potentially interfering with general syntactic flexibility. Without a baseline, you cannot distinguish between a model that has learned your data and a model that has simply forgotten how to speak generally.

Constructing the Regression Set

A rigorous evaluation suite must be bifurcated. You need two distinct types of held-out data, often referred to as a held out set, to ensure comprehensive coverage:

1. General Capability Benchmarks: These are standardized datasets like MMLU (Massive Multitask Language Understanding) or HellaSwag that test broad reasoning and general knowledge skills. These serve as your control group. If your fine-tuned model scores significantly lower here than the base model, you have a regression problem.
2. Domain-Specific Evaluation Set: This is a carefully curated, held-out set of examples from your target domain that the model has never seen. This measures the actual utility of the fine-tune.

Let’s assume you are fine-tuning Llama-3-8b for legal contract review. Your regression set includes 500 contracts for general legal reasoning (from MMLU-Law subsets) and 200 unseen contracts for specific clause extraction.

The Evaluation Pipeline

Research has shown that inconsistent prompting is a common error in LLM evaluation, leading to unreliable metrics. To ensure a valid comparison, the base model and the fine-tuned model must be evaluated under identical conditions. This includes using the exact same tokenizer, temperature, and system prompts.

Consider the following workflow using Python and Hugging Face transformers. The key is to instantiate both models and run them through the same inference loop.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
 
# Load base and fine-tuned models
base_model_name = "meta-llama/Meta-Llama-3-8B"
fine_tuned_path = "./output/llama3-legal-ft/checkpoint-1000"
 
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
# Use the same tokenizer for both to ensure consistent tokenization
tokenizer.pad_token = tokenizer.eos_token
 
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name, 
    torch_dtype=torch.float16, 
    device_map="auto"
)
ft_model = AutoModelForCausalLM.from_pretrained(
    fine_tuned_path, 
    torch_dtype=torch.float16, 
    device_map="auto"
)
 
def evaluate_model(model, tokenizer, prompt):
    inputs = tokenizer(prompt, return_tensors="pt", padding=True, truncation=True).to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=50, do_sample=False)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)
 
# Example prompt
prompt = "Extract the governing law clause from the following contract: [Contract Text]"
 
# Evaluate both
base_output = evaluate_model(base_model, tokenizer, prompt)
ft_output = evaluate_model(ft_model, tokenizer, prompt)
 
print(f"Base: {base_output}")
print(f"FT:   {ft_output}")

By running both models through this identical pipeline, you eliminate artifacts introduced by different tokenization strategies or generation parameters. The only variable is the weights.

Interpreting Delta Metrics

Once you have the outputs, you calculate the "delta" for each metric. Let’s say your results look like this:

• MMLU-Law Score: Base (72%) vs. Fine-Tune (68%) → Delta: -4%
• Clause Extraction Accuracy: Base (45%) vs. Fine-Tune (89%) → Delta: +44%

In this scenario, you have accepted a 4% regression in general legal reasoning to achieve a 44% gain in specific extraction accuracy. This is a successful trade-off. However, if the general score dropped by 15%, you might consider the fine-tune unusable for broader legal queries, limiting its scope strictly to clause extraction.

You should also track Perplexity on the held-out general set. Studies have shown that increased perplexity on held-out sets often correlates with higher hallucination rates. A significant increase in perplexity indicates the model is less confident in its general language understanding. This often correlates with the loss of nuance or hallucination rates.

Conclusion

Evaluating a fine-tune is an exercise in measuring trade-offs. There is no such thing as a "better" model in isolation; there is only a model better suited for a specific distribution. By treating the base model as the regression baseline, you gain visibility into the cost of specialization. Always report both the gain in domain performance and the loss in general capability. This transparency allows engineers to make informed decisions about whether the fine-tune is ready for production or if further regularization (such as LoRA rank reduction or learning rate adjustment) is required to preserve general knowledge.