Positional Encodings: Sinusoidal, RoPE, and ALiBi

Transformers are fundamentally permutation-invariant. Without explicit position signals, the model cannot distinguish between "The cat sat on the mat" and "The mat sat on the cat," because the set of input tokens is identical in both cases. To resolve this, positional encodings inject sequence order information into the embedding space. The evolution from Sinusoidal embeddings to Rotary Positional Embeddings (RoPE) and Attention with Linear Biases (ALiBi) reflects a deeper understanding of how attention mechanisms perceive distance, culminating in techniques like YARN that allow these models to generalize beyond their training context length.

The Problem of Permutation Invariance

Self-Attention mechanisms treat input sequences as sets rather than ordered lists. This permutation invariance means that reordering the input tokens does not change the output, which is problematic for language tasks where word order is critical. To fix this, positional encodings are added to token embeddings to provide a sense of sequence order.

The original solution, introduced in the seminal Transformer paper by Vaswani et al. (2017), uses fixed-frequency sine waves to generate distinct absolute positions via linear combinations. This approach maps each position to a unique vector in the embedding space, allowing the model to theoretically learn relative positions by processing these fixed coordinates. However, this method treats position as an absolute coordinate, which leads to significant issues when extending context length.

The Sinusoidal Foundation

The original Transformer architecture uses fixed sinusoidal functions to generate positional encodings. For a position $pos$ and dimension $i$, the encoding is defined as:

$PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d_{model}})$ $PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d_{model}})$

This approach maps each position to a unique vector in the embedding space. The mechanism relies on frequency: lower dimensions capture high-frequency oscillations, encoding fine-grained local differences (e.g., position 10 vs. 11), while higher dimensions capture low-frequency trends, encoding global structure (e.g., position 10 vs. 1000). Because sine and cosine are linearly independent, the model can theoretically learn to attend to relative positions by processing these fixed coordinates.

However, this method treats position as an absolute coordinate. If a model is trained on sequences of length 2048, the sinusoidal values for position 2049 are simply the next point in the wave. There is no guarantee that the geometric relationship between position 2048 and 2049 mirrors the relationship between 1 and 2, leading to poor extrapolation.

Rotary Positional Embeddings (RoPE)

RoPE, introduced by Su et al. (2021), addresses the extrapolation weakness of absolute positional encodings by encoding position through rotation rather than addition. Instead of adding a positional vector to the token embedding, RoPE rotates the query and key vectors in a complex plane.

For a token embedding vector $x_m$ at position $m$, RoPE applies a rotation matrix $R_m$ based on the dimension index $d$:

$R_m = \begin{bmatrix} \cos(m\theta_d) & -\sin(m\theta_d) \\ \sin(m\theta_d) & \cos(m\theta_d) \end{bmatrix}$

where $\theta_d = 10000^{-2d/d_{model}}$.

The critical mechanism here is how dot products behave under rotation. When computing attention scores, we calculate $q_m \cdot k_n$. After applying RoPE, the dot product becomes:

$\langle Q_m, K_n \rangle = \langle R_m Q, R_n K \rangle = \langle Q, R_{n-m} K \rangle$

This result depends only on the relative distance $n-m$, not on the absolute positions $m$ and $n$. This is a profound shift: the model learns relative distances intrinsically. If the model sees a distance of 5 during training, it will apply the same rotational penalty or boost regardless of whether that distance occurs at positions 1-6 or 1000-1005. While RoPE encodes relative distance well, its fixed frequency bases limit extrapolation beyond the training length, necessitating techniques like YARN for significant context extension.

Attention with Linear Biases (ALiBi)

While RoPE modifies the vectors multiplicatively, ALiBi modifies the attention scores additively. Instead of rotating queries and keys, ALiBi subtracts a linear bias from the attention matrix based on the distance between tokens:

$A_{i,j} = \text{softmax}\left(\frac{Q_i K_j^T}{\sqrt{d}} - m \cdot |i-j|\right)$

Here, $m$ is a slope parameter fixed before training as a geometric sequence, often different for each attention head. The mechanism encourages sparse attention: tokens far apart in the sequence receive lower attention scores due to the negative linear penalty. This induces an inductive bias towards local attention, which is often beneficial for language modeling.

ALiBi’s primary advantage is simplicity and zero-shot extrapolation. Because the bias is purely a function of distance, a model trained on length 2048 can process length 4096 by simply extending the distance calculation. There is no need to retrain or adjust frequency bases. However, ALiBi lacks the rich, dimension-wise interaction of RoPE. It treats all dimensions uniformly via the scalar slope, whereas RoPE allows different dimensions to encode different aspects of relative position through rotation angles. As noted by Press et al. (2021), this additive bias provides a robust alternative to multiplicative positional modifications.

Context Extension and YARN

Even with RoPE’s relative inductive bias, models still struggle with context extension. During training, the frequencies $\theta_d$ are fixed. When the context length increases significantly, the relative distances become larger, potentially pushing the rotational angles into regions where the model’s learned representations are less effective. This phenomenon is known as "frequency misalignment."

YARN (Yet Another RoPE extensioN), proposed by Peng et al. (2023), solves this by dynamically scaling the frequency bases to match the extended context length. The core mechanism involves scaling the base frequency parameter $\theta_d$ to maintain relative distance geometry, rather than simply interpolating values. If the original model was trained on length $L_{train}$ and we wish to extend to $L_{ext}$, YARN scales the base frequency:

$\theta_d' = \theta_d \times \left( \frac{L_{train}}{L_{ext}} \right)^{\frac{d}{d_{model}}}$

This scaling ensures that the maximum relative distance in the extended context maps to the same rotational angle as the maximum distance in the training context. By stretching the frequency spectrum, YARN preserves the relative geometric relationships that RoPE relies on, allowing the model to attend accurately to tokens at the very end of a much longer sequence.

Conclusion

Positional encodings are the bridge between static token sets and dynamic sequence understanding. Sinusoidal encodings provide absolute coordinates but fail at extrapolation. RoPE introduces rotational geometry to encode relative distance, offering superior generalization. ALiBi offers a simpler, additive bias for sparse attention. Finally, techniques like YARN refine RoPE by scaling its internal frequencies, enabling transformers to handle contexts far beyond their training distribution. For ML engineers, choosing among these methods involves balancing implementation complexity, extrapolation needs, and the specific attention patterns required by the task.