Linear algebra for representation learning
The Mathematics of Embeddings
An embedding maps each item of a discrete set (a word, a token, a product, a user) to a vector in . Training places the vectors so that geometry carries meaning: items used in similar contexts end up close together. This sheet covers the four pieces of mathematics behind that idea. It shows how a lookup is a matrix product, how closeness is measured, what the training objective factorises, and why high dimension changes how a similarity score should be read.
From a symbol to a vector
The life of an embedding
- Discrete symbols
- Learned parameters
- Vectors and operations on them
- Training signal
- Symbol to index — The tokenizer turns text into integer indices into a fixed vocabulary.
- Lookup as a product — Multiplying the one-hot vector by Eᵀ returns exactly one row of E.
- Use the vector — The vector is compared with others or passed to a model.
- Learn the matrix — The loss is differentiated with respect to E. Only the rows that were looked up receive a gradient.
Result derived in this sheet
Because has a single 1 in position , the product picks out row of . Libraries implement it as an indexed read rather than a multiplication. The product form still explains the gradient: is zero outside row .
| Symbol | Meaning | Unit |
|---|---|---|
| Embedding matrix, one row per vocabulary item, learned during training | ||
| One-hot vector of item : 1 in position , 0 elsewhere | ||
| Embedding of item | ||
| Vocabulary size (number of distinct items) | ||
| Embedding dimension, typically far smaller than | ||
| Training loss |
Measuring closeness
The cosine keeps only the angle between two vectors and discards their lengths. It equals 1 for vectors pointing the same way, 0 for orthogonal vectors and −1 for opposite ones. It is undefined if either vector is zero.
| Symbol | Meaning | Unit |
|---|---|---|
| Two embedding vectors | ||
| Their -th coordinates | ||
| Euclidean norm (length) of a vector |
| Criterion | Dot product | Cosine similarity | Euclidean distance |
|---|---|---|---|
| Formula | |||
| Range | |||
| A larger value means | More similar | More similar | Less similar |
| Depends on vector lengths | Yes: long vectors score high | No: direction only | Yes |
| Typical use | Attention scores, output logits | Semantic search when norms vary | Clustering and nearest neighbours on normalised vectors |
Three ways to compare two embeddings
Dot product
- Formula
- Range
- A larger value means
- More similar
- Depends on vector lengths
- Yes: long vectors score high
- Typical use
- Attention scores, output logits
Cosine similarity
- Formula
- Range
- A larger value means
- More similar
- Depends on vector lengths
- No: direction only
- Typical use
- Semantic search when norms vary
Euclidean distance
- Formula
- Range
- A larger value means
- Less similar
- Depends on vector lengths
- Yes
- Typical use
- Clustering and nearest neighbours on normalised vectors
VerdictOnce every vector is scaled to unit length, all three give the same ranking of neighbours (see the identity below). They differ only on unnormalised vectors.
Cosine similarities of four toy vectors
Result derived in this sheet
Expand the squared norm. With unit vectors the squared distance is a decreasing function of the cosine. Normalising first therefore makes nearest-neighbour search by distance, by cosine and by dot product interchangeable. This is why vector databases often store normalised embeddings.
| Symbol | Meaning | Unit |
|---|---|---|
| Two embeddings scaled to unit length |
Where the vectors come from
This quantity is maximised over all (word, context) pairs observed within a sliding window. The first term pushes the dot product of a real pair up. The second pushes down the dot products with random negative contexts. In the original paper is the unigram distribution raised to the power 3/4 2. Two matrices are learned: word vectors and context vectors .
| Symbol | Meaning | Unit |
|---|---|---|
| A centre word and a word observed in its context window | ||
| Word (input) embedding of | ||
| Context (output) embedding of | ||
| Logistic function | ||
| Number of negative samples per observed pair | ||
| Noise distribution from which negative contexts are drawn |
Sources:2
Geometry in high dimension
Result derived in this sheet
For independent and uniform on the unit sphere, rotation invariance lets us fix , so . Since and all coordinates play the same role, . In the typical chance cosine is about , so most pairs of random directions are close to orthogonal. Trained embeddings are not uniform on the sphere: contextual embeddings in particular have been found to be strongly anisotropic, crowding into a narrow cone 6. The chance baseline for a given model must therefore be measured, not assumed.
| Symbol | Meaning | Unit |
|---|---|---|
| Independent random unit vectors, uniformly distributed on the sphere | ||
| Dimension of the space | ||
| First coordinate of |
Sources:6
Check and explore
Dot products turned into attention weights
Synthetic data — seed 7Attention weights of a single head. Embeddings (one vector per word plus a positional term) and the and projections are drawn from the seed: the map shows the mechanism, not what a trained model has learned. is a simplified relative-position bias; and give the standard formula.
Interactive heatmap of single-head attention weights between the words of a short sentence. Each row is a query word and each column a key word. Cell brightness is the softmax of the scaled dot product between projected embeddings. Sliders control the temperature, the vector dimension and a locality bias.
- Mean row entropy (normalised)
- 0.92
- Mean weight on itself
- 0.208
- Causal mask
- no
Sources:7
QuestionTwo unit-length embeddings are at Euclidean distance from each other. What is their cosine similarity?
- A1
- B0
- C−1
- D
Show the answer
- A — Incorrect. A cosine of 1 means identical directions, hence distance 0.
- B — Correct. From : , so . The vectors are orthogonal.
- C — Incorrect. Opposite unit vectors are at distance 2, not .
- D — Incorrect. That would give a squared distance of , i.e. a distance of about 0.77.
References
- (2013). Efficient Estimation of Word Representations in Vector Space. arXiv preprint. arxiv.org/abs/1301.3781 unverified
- (2013). Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems 26 (NIPS 2013). unverified
- (2014). Neural Word Embedding as Implicit Matrix Factorization. Advances in Neural Information Processing Systems 27 (NIPS 2014). unverified
- (2014). GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). unverified
- (2016). Issues in evaluating semantic spaces using word analogies. Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP. unverified
- (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP). unverified
- (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017). arxiv.org/abs/1706.03762 unverified