Function-Calling Fine-Tunes: Teaching a Small Model Your Tools

Small language models (SLMs) like Llama-3-8B, Mistral-7B, or Phi-3 are increasingly attractive for production environments due to their lower inference costs and reduced memory footprint. However, they frequently exhibit a critical weakness in tool use: they struggle to generate valid, structured function arguments when asked to perform specific, non-standard actions. While large models (70B+) can often reason through complex schemas via few-shot prompting, SLMs tend to hallucinate argument structures or fail to output valid JSON, breaking the pipeline.

This failure is not due to a lack of general intelligence but a lack of exposure to the specific syntactic patterns of function calling. By fine-tuning these models on mined execution traces, engineers can teach them the precise mapping between natural language intent and structured tool inputs. This guide details the mechanism of mining these traces and structuring the fine-tuning process to achieve reliable tool use.

The Failure Mode of Zero-Shot SLMs

When an SLM is prompted to call a function, it must perform two distinct cognitive tasks: semantic understanding (what does the user want?) and syntactic generation (how do I format this as JSON?). In zero-shot scenarios, SLMs often prioritize semantic coherence over syntactic validity. They may produce a plausible-sounding narrative but fail to output a parseable JSON object, or they may output valid JSON but with incorrect keys or types.

This happens because the pre-training data of most open-weight models is dominated by natural language text, code repositories, and general instruction-following examples. Structured tool-use patterns are rare in this corpus. Consequently, the model’s probability distribution for generating a closing brace } or a specific key like user_id is not sufficiently optimized for the constraints of your specific API schema. Effective tool use training requires addressing this gap by exposing the model to the specific syntactic structures it needs to master.

Mining Execution Traces

The most effective way to generate training data for function calling is not to manually write prompts, but to mine execution traces from your existing production systems. If you have a working system that uses a larger model or a rule-based parser to handle tool calls, you likely have a goldmine of successful interactions.

The mechanism here is reverse-engineering the ground truth. You extract pairs of (user_query, successful_api_call) from your logs. An execution trace typically includes:

1. The raw user input.
2. The selected function name (e.g., get_weather).
3. The exact JSON arguments passed to the API (e.g., {"location": "London", "unit": "celsius"}).

By collecting hundreds or thousands of these traces, you create a dataset that reflects the specific jargon, edge cases, and formatting quirks of your application. This approach is superior to synthetic data generation because it captures real-world ambiguity that synthetic prompts often miss. Tools like LangChain’s LangSmith or Arize’s Phoenix can help automate the extraction and labeling of these traces, ensuring that even a small model can learn from high-quality, real-world examples.

Structuring the Fine-Tuning Data

Once you have mined your traces, you must format them for instruction tuning. The standard format is JSONL, where each line is a dictionary containing the conversation history and the assistant’s response. For function calling, the structure is critical.

Consider a trace where a user asks, "What's the weather in London?" and the system calls get_weather. The fine-tuning example should look like this:

{
  "messages": [
    {"role": "user", "content": "What's the weather in London?"},
    {"role": "assistant", "content": null, "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"location\": \"London\", \"unit\": \"celsius\"}"}}]}
  ]
}

Notice that the arguments field is a stringified JSON object. This is a crucial detail. Many SLMs struggle if the arguments are provided as a nested Python object in the training data; they need to learn to generate the string. The model must learn that the output of the assistant role is not just text, but a structured object containing tool_calls.

You should also include negative examples or "near-misses" if available. For instance, if your system previously rejected a malformed JSON input, include the corrected version as the target response. This teaches the model what not to do by showing the correction. The quality of this trace mining directly impacts the model's ability to generalize.

The Training Loop and Validation

Fine-tuning an SLM for function calling is computationally cheaper than training from scratch. Using techniques like QLoRA (Quantized Low-Rank Adaptation), you can fine-tune a 7B parameter model on a single GPU with 24GB of VRAM. The mechanism of QLoRA involves freezing the pre-trained weights and injecting small, trainable rank decomposition matrices into the linear layers of the transformer. This allows the model to adapt to the new syntax without overwriting its general language capabilities (Dettmers et al., 2023).

However, standard metrics like perplexity are insufficient for evaluating function-calling models. A model might have low perplexity but still output invalid JSON. You must implement a custom evaluation pipeline that:

1. Parses the model’s output.
2. Checks if it is valid JSON.
3. Validates the JSON against your API schema (e.g., using pydantic).

The key metric is the parsing success rate. Aim for a parsing success rate above 95% on your validation set before deploying. If the rate is lower, you likely need more diverse training data, particularly covering edge cases like missing arguments or ambiguous user intents.

Operationalizing the Model

Once fine-tuned, integrate the model into your agent loop. The workflow remains similar to zero-shot function calling: the model receives the user prompt and the list of available tools, then outputs a JSON object with the function name and arguments. Your backend parses this output and executes the function.

The primary advantage of fine-tuning is reliability. In high-volume scenarios, the cost of API calls to large models adds up, and the latency of complex prompting increases. A fine-tuned SLM can respond faster and with higher accuracy, reducing the need for retry loops when the initial output is malformed.

While fine-tuning is powerful, it is not a silver bullet. If your tool schema changes frequently, maintaining a fine-tuned model becomes cumbersome. In such cases, consider using a hybrid approach: fine-tune for the most common 80% of tool calls, and fall back to a larger model or few-shot prompting for the remaining 20% of complex, rare cases. This balances cost, latency, and reliability.

Conclusion

Teaching a small model to use tools effectively is not about finding a smarter architecture; it’s about providing the right data. By mining execution traces and fine-tuning on structured JSON outputs, you align the model’s probability distribution with the syntactic requirements of your API. This process transforms an unreliable zero-shot predictor into a robust tool-use agent, enabling efficient, low-cost deployment of advanced AI features.