How Embeddings Encode Meaning

To understand retrieval engineering, one must abandon the intuition that embeddings are static containers of definition. A vector does not hold a dictionary entry; it holds a coordinate in a high-dimensional manifold where proximity implies semantic relationship. This spatial arrangement is not innate. It is the result of rigorous geometric optimization, specifically through contrastive learning, which sculpts the space by enforcing constraints on positive and negative examples. However, this process introduces statistical distortions known as anisotropy, which can reduce the utility of cosine similarity for retrieval by compressing the angular distribution, making it harder to distinguish between relevant and irrelevant documents.

The Geometry of Distributed Representation

In natural language processing, meaning is encoded via distributed representation. Instead of assigning a unique ID to a word (one-hot encoding), models project tokens into a dense vector space where each dimension captures a latent feature. The mechanism here is contextual: the meaning of "bank" shifts based on its neighbors, moving its vector position within the space.

Consider two sentences: "I deposited money at the bank" and "I sat on the river bank." In a well-trained embedding space, the vector for the first instance will be close to vectors representing "finance," "deposit," and "account." The vector for the second instance will be close to "river," "shore," and "water." The distance between these two "bank" vectors is large, reflecting their semantic divergence.

This proximity is measured using cosine similarity, which calculates the cosine of the angle between two vectors. Unlike Euclidean distance, which measures the straight-line path between points, cosine similarity focuses on orientation. If two vectors point in the same direction, their cosine similarity is 1, regardless of their magnitude. This is critical because the magnitude of an embedding vector often correlates with term frequency or general informativeness, not just semantic alignment. By normalizing vectors to unit length, we isolate the directional relationship, ensuring that "machine learning" and "ML" align closely if they share context, even if one appears more frequently in the training data.

Contrastive Learning: Sculpting the Manifold

The structure of this space is not random. It is forged by contrastive learning objectives. The core mechanism is simple: the model is presented with pairs of text and asked to maximize the similarity of positive pairs while minimizing the similarity of negative pairs.

In a typical training scenario, the model sees a query "What is the capital of France?" and a positive document "Paris is the capital of France." It also sees a negative document "London is the capital of the UK." The loss function penalizes the model if the vector for the query is far from the positive document or close to the negative document.

This process creates a manifold where semantically related items cluster tightly. The mechanism relies on the gradient descent algorithm adjusting the weights of the neural network to reduce the distance between positive pairs and increase it for negatives. Over millions of iterations, this forces the space to organize itself. Concepts that frequently appear in similar contexts end up near each other. Linear relationships in vector space, such as the famous "king minus man plus woman" example, are an emergent property of distributed representations trained with various objectives, including Word2Vec and GloVe. Contrastive learning is one method that can enhance this structure by explicitly pulling related items together in the optimization landscape.

The Anisotropy Problem

Despite the elegance of contrastive learning, embedding spaces suffer from a structural flaw known as anisotropy. In an ideal isotropic space, vectors would be uniformly distributed in all directions, like stars in a sphere. In reality, embeddings tend to cluster in a narrow cone within the vector space, pointing in similar directions with little variance in angle.

This phenomenon occurs due to the high dimensionality and unbalanced vocabulary size relative to embedding dimensions. As noted in research such as Ethayarajh's (2019) study on anisotropy in contextual embeddings and Li et al.'s (2020) work introducing BERT-flow, anisotropy arises from the geometry of the loss landscape and vocabulary imbalance, causing embeddings to cluster around a mean vector (norm direction). This clustering is not primarily driven by softmax normalization, although softmax plays a role in transformer architectures; the fundamental issue is that the optimization process favors solutions where vectors align with the average direction of the data distribution.

As a result, the angles between most pairs of vectors become small, leading to high cosine similarities even for unrelated texts. For example, if you compute the cosine similarity between "The cat sat on the mat" and "The dog played in the park," you might get a surprisingly high score (e.g., 0.85) simply because both vectors point in the general direction of the dominant mode of the distribution. This makes it difficult to distinguish between relevant and irrelevant documents in retrieval tasks. The metric does not break mathematically, but its discriminative power is degraded because the space is not spread out; it is collapsed.

Mitigating Anisotropy

To restore meaningful distance metrics, engineers must apply corrections. The most common technique is L2 normalization, which projects vectors onto the surface of a unit hypersphere. This ensures that all vectors have the same magnitude, making cosine similarity equivalent to dot product. However, normalization alone does not fix anisotropy; it merely standardizes the scale.

More advanced techniques involve modifying the training objective or post-processing the vectors. For instance, SimCSE (Simple Contrastive Learning of Sentence Embeddings) uses dropout as a form of data augmentation to create positive pairs from the same sentence, forcing the model to learn robust representations that are invariant to noise. Additionally, some approaches add a bias term to the embeddings or use projection heads that explicitly decorrelate the dimensions.

Another practical mitigation is to adjust the similarity threshold. Since anisotropic spaces produce higher baseline similarities, the threshold for considering a match "relevant" must be raised. Alternatively, using inner product similarity instead of cosine similarity can sometimes help, as it retains magnitude information which may correlate with topic specificity.

Conclusion

Embeddings encode meaning by mapping semantic relationships into geometric proximity. This mapping is shaped by contrastive learning, which pulls related items together and pushes unrelated ones apart. However, the inherent anisotropy of these spaces can distort similarity metrics, leading to false positives in retrieval. Understanding these mechanisms allows engineers to choose the right normalization techniques and similarity metrics, ensuring that the vector space accurately reflects the semantic structure of the data. By recognizing the geometry behind the vectors, we can build more reliable retrieval systems that truly understand the nuances of language.