Model Merging: SLERP, TIES, and DARE

The Geometry of Merging: SLERP, TIES, and DARE

Most machine learning engineers view model training as a monolithic event: data flows in, gradients backpropagate, and weights update. However, a growing subset of the community treats weights not as immutable endpoints, but as malleable artifacts that can be combined algebraically. This practice, known as model merging, allows engineers to create a "super-model" from several specialized fine-tunes without incurring the compute cost of joint training.

The core challenge in merging is that neural network weights are high-dimensional vectors. Simply adding them together often results in catastrophic interference, where the knowledge from one model cancels out the other. To merge effectively, we must understand the geometric and statistical properties of these weight spaces. Three techniques—SLERP, TIES, and DARE—address this problem through distinct mechanisms: spherical geometry, conflict resolution, and sparse expectation preservation.

SLERP: Preserving Norms on the Manifold

Linear interpolation (LERP) is the default assumption for mixing values. If you have two models, $M_1$ and $M_2$, a simple merge might look like:

merged_weights = alpha * M1.weights + (1 - alpha) * M2.weights

While this works for simple scalars, it fails in high-dimensional weight spaces because it does not preserve the norm (magnitude) of the weight vectors. In transformer layers, the scale of embeddings and attention heads is critical for stability. Linear interpolation shrinks the magnitude of vectors as they move between two points, potentially degrading the model's capacity.

Spherical Linear Interpolation (SLERP) solves this by treating weight vectors as points on a hypersphere. Instead of moving through the interior of the sphere (the chord), SLERP moves along the arc. This preserves the norm of the weights throughout the interpolation process.

For two weight matrices $W_1$ and $W_2$, SLERP computes the interpolated weight $W_{interp}$ using the angle $\omega$ between the two vectors:

$W_{interp} = \frac{\sin((1-\theta)\omega)}{\sin(\omega)} W_1 + \frac{\sin(\theta\omega)}{\sin(\omega)} W_2$

where $\theta$ is the interpolation factor (equivalent to $\alpha$ in LERP).

This mechanism is particularly useful when merging two models that have similar architectures but were trained on different domains or tasks. By preserving the magnitude, SLERP ensures that the resulting model retains the representational capacity of the original components. It is important to note that standard SLERP is primarily a binary operation. While it defines a unique geodesic path between two points, combining three or more models simultaneously requires either pairwise application of SLERP or specific extensions that define multi-way geodesics. Consequently, for direct multi-model merging, other methods like TIES are often preferred for their native support of multiple inputs.

TIES: Resolving Conflict in Multi-Model Merging

When moving beyond two models, the complexity increases. If you merge three models fine-tuned for different tasks, their weight updates may point in conflicting directions. For example, Model A might increase a specific weight to recognize cats, while Model B decreases it to recognize dogs. Simple averaging would result in a neutral weight that recognizes neither well.

The TIES (TRIM, Elect Sign & Merge) method addresses this by introducing a pruning and sign-resolution step before summation. The process involves three key stages:

1. Task Vector Extraction: Compute the difference between the fine-tuned model and the base model ($W_{ft} - W_{base}$). This isolates the specific changes made for each task.
2. Pruning: Remove small updates that are likely noise. This reduces the parameter space to only the most significant changes. Pruning thresholds can be set globally or based on the magnitude of updates relative to the base model's weights.
3. Sign Resolution: For each weight position across all models, determine the dominant sign. If most models agree on increasing a weight, the merged weight increases. If they disagree, the conflicting updates are suppressed or averaged based on magnitude.

This approach relies on the observation that while different tasks may affect different parts of the network, there is often redundancy. By resolving sign ambiguity, TIES prevents destructive interference. The result is a model that retains the capabilities of all input models, provided they do not fundamentally contradict each other. Sign resolution heuristics typically involve a vote or a weighted average based on the confidence (magnitude) of each task vector's contribution.

DARE: Sparse Merging via Expected Value

A newer and highly efficient approach is DARE (Drop And REscale). Introduced by Yu et al., DARE operates on the principle that large language models contain significant redundancy. Many weights are close to zero or have minimal impact on the final output. DARE leverages this by treating the merge as a sparse sampling problem.

Instead of merging all weights, DARE randomly drops a fraction of the non-zero weights from the input models. To maintain the expected value of the tensor, it rescales the remaining weights by dividing them by the probability of survival.

# Conceptual implementation of DARE
def dare_merge(models, drop_rate):
    merged = models[0].copy()
    for model in models[1:]:
        # Create a mask where True means keep, False means drop
        mask = torch.rand_like(merged.weight) > drop_rate
        
        # Rescale kept values to maintain expected sum
        merged.weight[mask] += model.weight[mask] / (1 - drop_rate)
        
    return merged

The key insight here is statistical. If you drop 90% of the weights and scale the rest by 10x, the expected value of the tensor remains unchanged. This allows DARE to perform merges with extremely high sparsity in the task vectors, reducing the impact of noisy updates. However, DARE results in a DENSE model; the parameter count and memory footprint remain identical to the input models. Any latency gains would come from subsequent pruning or quantization steps applied to the merged result, not from the merge itself.

DARE is particularly effective when merging models with identical architectures, as it relies on the alignment of weight positions across models. Practical applications often use drop rates between 0.9 and 0.99 for large language models. Unlike SLERP, which is best for binary merges, or TIES, which handles conflict resolution explicitly, DARE excels at preserving the expected value of the combined task vectors, often outperforming simple averaging in multi-model scenarios where noise accumulation is a concern.

Practical Considerations and Tradeoffs

Choosing between SLERP, TIES, and DARE depends on the specific use case. SLERP is ideal for fine-grained interpolation between two similar models. TIES is superior for merging multiple models with distinct, potentially conflicting tasks. DARE is best for reducing the impact of noise in multi-model merges while maintaining the original model density.

It is important to note that these methods assume architectural alignment. You cannot merge a GPT-2 model with a Llama-2 model directly because their weight dimensions and layer structures differ. Tools like mergekit automate these processes, handling the alignment and providing a unified interface for these algorithms.

In conclusion, model merging is not just a shortcut; it is a rigorous manipulation of weight space. By understanding the geometry of SLERP, the conflict resolution of TIES, and the statistical properties of DARE, engineers can compose specialized models into powerful, efficient systems without the overhead of full retraining.

Common Pitfalls

1. Architectural Misalignment: Attempting to merge models with different hidden sizes, layer counts, or tokenizers will result in errors or meaningless outputs. Always verify that models share the same architecture before merging.
2. Catastrophic Interference in Simple Averaging: Using simple linear interpolation (LERP) without considering norm preservation or sign resolution can lead to catastrophic interference, where the merged model performs worse than its components.
3. Over-Reliance on Single Metrics: Evaluating merged models solely on metrics like perplexity can be misleading. Perplexity does not always correlate with downstream task performance. Always evaluate merged models on specific downstream tasks relevant to your use case.

Practical Takeaways

1. SLERP for Binary Norm Preservation: Use SLERP when merging two models to maintain the magnitude of weight vectors, ensuring stability in attention heads and embeddings.
2. TIES for Multi-Model Conflict Resolution: Use TIES when combining three or more models to resolve sign conflicts and prune noisy updates, preserving the capabilities of all input models.
3. DARE for Efficient Density-Preserving Merges: Use DARE when merging multiple models with identical architectures to reduce noise impact while maintaining the original model size, suitable for subsequent pruning or quantization.

FAQ

Can I merge models with different hidden sizes? No, model merging requires architectural alignment. Models must have the same number of layers, hidden sizes, and attention heads to ensure weight correspondence.

Does model merging improve generalization? Yes, merging specialized models can often improve generalization by combining diverse knowledge sources. However, this depends on the compatibility of the models and the merging technique used.

How does mergekit simplify these processes? mergekit provides a user-friendly interface for applying various merging algorithms (SLERP, TIES, DARE, etc.) without manual implementation. It handles architectural alignment, weight normalization, and provides a consistent API for experimenting with different merge strategies.