Aller au contenu

Machine learningNatural language processingLinear algebraExplicationIntermédiaire

Par · Publiée le · 10 min de lecture

Sommaire
  1. From a symbol to a vector
  2. Measuring closeness
  3. Where the vectors come from
  4. Geometry in high dimension
  5. Check and explore

Après cette fiche, vous saurez

  • Write an embedding lookup as the product of a one-hot vector and an embedding matrix
  • Choose between dot product, cosine similarity and Euclidean distance, and know when they rank neighbours identically
  • State the skip-gram negative-sampling objective and its link to pointwise mutual information
  • Interpret a similarity score against the near-orthogonality of random vectors in high dimension
  • Recognise the limits of word-analogy demonstrations

Formulaire

Les équations de la fiche, d’un coup d’œil

  1. 01Embedding lookupei=E⊤xi=(Ei,1,Ei,2,…,Ei,d)⊤\mathbf{e}_i = E^{\top}\mathbf{x}_i = \big(E_{i,1}, E_{i,2}, \dots, E_{i,d}\big)^{\top}
  2. 02Cosine similaritycos⁡(u,v)=u⊤v∥u∥2 ∥v∥2=∑k=1dukvk∑k=1duk2 ∑k=1dvk2\cos(\mathbf{u},\mathbf{v}) = \frac{\mathbf{u}^{\top}\mathbf{v}}{\lVert\mathbf{u}\rVert_2\,\lVert\mathbf{v}\rVert_2} = \frac{\sum_{k=1}^{d} u_k v_k}{\sqrt{\sum_{k=1}^{d} u_k^2}\,\sqrt{\sum_{k=1}^{d} v_k^2}}
  3. 03Distance and cosine on unit vectors∥u−v∥22=∥u∥2+∥v∥2−2 u⊤v  =  2−2cos⁡(u,v)if ∥u∥=∥v∥=1\lVert\mathbf{u}-\mathbf{v}\rVert_2^2 = \lVert\mathbf{u}\rVert^2 + \lVert\mathbf{v}\rVert^2 - 2\,\mathbf{u}^{\top}\mathbf{v} \;=\; 2 - 2\cos(\mathbf{u},\mathbf{v}) \quad \text{if } \lVert\mathbf{u}\rVert = \lVert\mathbf{v}\rVert = 1
  4. 04Skip-gram with negative sampling (word2vec)ℓ(w,c)=log⁡σ ⁣(uc⊤vw)+∑j=1kEcj∼Pn ⁣[log⁡σ ⁣(−ucj⊤vw)]\ell(w,c) = \log \sigma\!\left(\mathbf{u}_c^{\top}\mathbf{v}_w\right) + \sum_{j=1}^{k} \mathbb{E}_{c_j \sim P_n}\!\left[\log \sigma\!\left(-\mathbf{u}_{c_j}^{\top}\mathbf{v}_w\right)\right]
  5. 05Random directions are nearly orthogonalE[cos⁡(u,v)]=0,Var⁡[cos⁡(u,v)]=1d\mathbb{E}\left[\cos(\mathbf{u},\mathbf{v})\right] = 0, \qquad \operatorname{Var}\left[\cos(\mathbf{u},\mathbf{v})\right] = \frac{1}{d}

Linear algebra for representation learning

The Mathematics of Embeddings

An embedding maps each item of a discrete set (a word, a token, a product, a user) to a vector in Rd\mathbb{R}^d. Training places the vectors so that geometry carries meaning: items used in similar contexts end up close together. This sheet covers the four pieces of mathematics behind that idea. It shows how a lookup is a matrix product, how closeness is measured, what the training objective factorises, and why high dimension changes how a similarity score should be read.

From a symbol to a vector

The life of an embedding

Flow diagram from left to right. A token goes through a tokenizer to an integer index, then to a one-hot vector. Multiplying the one-hot vector by the embedding matrix selects one row, giving a dense vector. The vector goes to a similarity computation and to a downstream model. The model's predictions produce a training loss, and a dashed arrow from the loss returns to the embedding matrix to show that gradients update it.DISCRETE SYMBOLSVECTOR SPACETokene.g. "cat"Index iposition in a vocabulary ofsize VOne-hot vector xx ∈ {0,1}^V, a single 1Embedding matrix EV × d, learnedDense vector ee ∈ ℝᵈSimilaritydot product, cosine, distanceDownstream modele.g. attention layersTraining losse.g. predict the contexttokenizerencodemultiply by Eᵀselects row icompareinputpredictionsgradients update E
  • Discrete symbols
  • Learned parameters
  • Vectors and operations on them
  • Training signal
  1. Symbol to index — The tokenizer turns text into integer indices into a fixed vocabulary.
  2. Lookup as a product — Multiplying the one-hot vector by Eᵀ returns exactly one row of E.
  3. Use the vector — The vector is compared with others or passed to a model.
  4. Learn the matrix — The loss is differentiated with respect to E. Only the rows that were looked up receive a gradient.
A token becomes an index, the index selects one row of a learned matrix, and the resulting vector is compared or fed to a model. The training loss sends gradients back into the matrix.
Embedding lookup
ei=E⊤xi=(Ei,1,Ei,2,…,Ei,d)⊤\mathbf{e}_i = E^{\top}\mathbf{x}_i = \big(E_{i,1}, E_{i,2}, \dots, E_{i,d}\big)^{\top}

Result derived in this sheet

Because xi\mathbf{x}_i has a single 1 in position ii, the product picks out row ii of EE. Libraries implement it as an indexed read rather than a multiplication. The product form still explains the gradient: ∂L/∂E=xi (∂L/∂ei)⊤\partial \mathcal{L}/\partial E = \mathbf{x}_i\,(\partial \mathcal{L}/\partial \mathbf{e}_i)^{\top} is zero outside row ii.

Variables
SymbolMeaningUnit
EEEmbedding matrix, one row per vocabulary item, learned during trainingRV×d\mathbb{R}^{V\times d}
xi\mathbf{x}_iOne-hot vector of item ii: 1 in position ii, 0 elsewhere{0,1}V\{0,1\}^V
ei\mathbf{e}_iEmbedding of item iiRd\mathbb{R}^d
VVVocabulary size (number of distinct items)
ddEmbedding dimension, typically far smaller than VV
L\mathcal{L}Training loss

Measuring closeness

Cosine similarity
cos⁡(u,v)=u⊤v∥u∥2 ∥v∥2=∑k=1dukvk∑k=1duk2 ∑k=1dvk2\cos(\mathbf{u},\mathbf{v}) = \frac{\mathbf{u}^{\top}\mathbf{v}}{\lVert\mathbf{u}\rVert_2\,\lVert\mathbf{v}\rVert_2} = \frac{\sum_{k=1}^{d} u_k v_k}{\sqrt{\sum_{k=1}^{d} u_k^2}\,\sqrt{\sum_{k=1}^{d} v_k^2}}

The cosine keeps only the angle between two vectors and discards their lengths. It equals 1 for vectors pointing the same way, 0 for orthogonal vectors and −1 for opposite ones. It is undefined if either vector is zero.

Variables
SymbolMeaningUnit
u,v\mathbf{u},\mathbf{v}Two embedding vectorsRd\mathbb{R}^d
uk,vku_k, v_kTheir kk-th coordinates
∥⋅∥2\lVert\cdot\rVert_2Euclidean norm (length) of a vector
Three ways to compare two embeddings
CriterionDot productCosine similarityEuclidean distance
Formulau⊤v\mathbf{u}^{\top}\mathbf{v}u⊤v / (∥u∥ ∥v∥)\mathbf{u}^{\top}\mathbf{v}\,/\,(\lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert)∥u−v∥2\lVert\mathbf{u}-\mathbf{v}\rVert_2
Range(−∞,∞)(-\infty, \infty)[−1,1][-1, 1][0,∞)[0, \infty)
A larger value meansMore similarMore similarLess similar
Depends on vector lengthsYes: long vectors score highNo: direction onlyYes
Typical useAttention scores, output logitsSemantic search when norms varyClustering and nearest neighbours on normalised vectors

Three ways to compare two embeddings

Dot product

Formula
u⊤v\mathbf{u}^{\top}\mathbf{v}
Range
(−∞,∞)(-\infty, \infty)
A larger value means
More similar
Depends on vector lengths
Yes: long vectors score high
Typical use
Attention scores, output logits

Cosine similarity

Formula
u⊤v / (∥u∥ ∥v∥)\mathbf{u}^{\top}\mathbf{v}\,/\,(\lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert)
Range
[−1,1][-1, 1]
A larger value means
More similar
Depends on vector lengths
No: direction only
Typical use
Semantic search when norms vary

Euclidean distance

Formula
∥u−v∥2\lVert\mathbf{u}-\mathbf{v}\rVert_2
Range
[0,∞)[0, \infty)
A larger value means
Less similar
Depends on vector lengths
Yes
Typical use
Clustering and nearest neighbours on normalised vectors

VerdictOnce every vector is scaled to unit length, all three give the same ranking of neighbours (see the identity below). They differ only on unnormalised vectors.

Cosine similarities of four toy vectors

A 4 by 4 heatmap of cosine similarities between toy vectors for cat, dog, car and truck. The diagonal is 1. Cat and dog score 0.96, and car and truck also score 0.96. Cross pairs score low: cat–car 0.10, cat–truck 0.18, dog–car 0.18, dog–truck 0.31. Two bright blocks show the two clusters.SYNTHETIC DATAcatdogcartruckcatdogcartruck1.000.960.100.180.961.000.180.310.100.181.000.960.180.310.961.000.101.00
Synthetic: computed exactly from hand-picked 3-d vectors cat = (3, 1, 0), dog = (3, 2, 0), car = (0, 1, 3), truck = (0, 2, 3). The vectors are not from any trained model.
Distance and cosine on unit vectors
∥u−v∥22=∥u∥2+∥v∥2−2 u⊤v  =  2−2cos⁡(u,v)if ∥u∥=∥v∥=1\lVert\mathbf{u}-\mathbf{v}\rVert_2^2 = \lVert\mathbf{u}\rVert^2 + \lVert\mathbf{v}\rVert^2 - 2\,\mathbf{u}^{\top}\mathbf{v} \;=\; 2 - 2\cos(\mathbf{u},\mathbf{v}) \quad \text{if } \lVert\mathbf{u}\rVert = \lVert\mathbf{v}\rVert = 1

Result derived in this sheet

Expand the squared norm. With unit vectors the squared distance is a decreasing function of the cosine. Normalising first therefore makes nearest-neighbour search by distance, by cosine and by dot product interchangeable. This is why vector databases often store normalised embeddings.

Variables
SymbolMeaningUnit
u,v\mathbf{u},\mathbf{v}Two embeddings scaled to unit lengthRd\mathbb{R}^d

Where the vectors come from

Skip-gram with negative sampling (word2vec)
ℓ(w,c)=log⁡σ ⁣(uc⊤vw)+∑j=1kEcj∼Pn ⁣[log⁡σ ⁣(−ucj⊤vw)]\ell(w,c) = \log \sigma\!\left(\mathbf{u}_c^{\top}\mathbf{v}_w\right) + \sum_{j=1}^{k} \mathbb{E}_{c_j \sim P_n}\!\left[\log \sigma\!\left(-\mathbf{u}_{c_j}^{\top}\mathbf{v}_w\right)\right]

This quantity is maximised over all (word, context) pairs observed within a sliding window. The first term pushes the dot product of a real pair up. The second pushes down the dot products with kk random negative contexts. In the original paper PnP_n is the unigram distribution raised to the power 3/4 2. Two matrices are learned: word vectors v\mathbf{v} and context vectors u\mathbf{u}.

Variables
SymbolMeaningUnit
w,cw, cA centre word and a word observed in its context window
vw\mathbf{v}_wWord (input) embedding of wwRd\mathbb{R}^d
uc\mathbf{u}_cContext (output) embedding of ccRd\mathbb{R}^d
σ(z)\sigma(z)Logistic function 1/(1+e−z)1/(1+e^{-z})
kkNumber of negative samples per observed pair
PnP_nNoise distribution from which negative contexts are drawn

Sources:2

Sources:34

Geometry in high dimension

Random directions are nearly orthogonal
E[cos⁡(u,v)]=0,Var⁡[cos⁡(u,v)]=1d\mathbb{E}\left[\cos(\mathbf{u},\mathbf{v})\right] = 0, \qquad \operatorname{Var}\left[\cos(\mathbf{u},\mathbf{v})\right] = \frac{1}{d}

Result derived in this sheet

For u,v\mathbf{u},\mathbf{v} independent and uniform on the unit sphere, rotation invariance lets us fix v=(1,0,…,0)\mathbf{v} = (1,0,\dots,0), so cos⁡=u1\cos = u_1. Since ∑kuk2=1\sum_k u_k^2 = 1 and all coordinates play the same role, E[u12]=1/d\mathbb{E}[u_1^2] = 1/d. In d=300d = 300 the typical chance cosine is about 1/300≈0.061/\sqrt{300} \approx 0.06, so most pairs of random directions are close to orthogonal. Trained embeddings are not uniform on the sphere: contextual embeddings in particular have been found to be strongly anisotropic, crowding into a narrow cone 6. The chance baseline for a given model must therefore be measured, not assumed.

Variables
SymbolMeaningUnit
u,v\mathbf{u},\mathbf{v}Independent random unit vectors, uniformly distributed on the sphereRd\mathbb{R}^d
ddDimension of the space
u1u_1First coordinate of u\mathbf{u}

Sources:6

Sources:15

Check and explore

Dot products turned into attention weights

Synthetic data — seed 7

Attention weights softmax⁡((qi⊤kj/d−λ ∣i−j∣)/τ)\operatorname{softmax}\big((q_i^{\top} k_j/\sqrt{d} - \lambda\,|i-j|)/\tau\big) of a single head. Embeddings (one vector per word plus a positional term) and the QQ and KK projections are drawn from the seed: the map shows the mechanism, not what a trained model has learned. λ\lambda is a simplified relative-position bias; τ=1\tau = 1 and λ=0\lambda = 0 give the standard formula.

Interactive heatmap of single-head attention weights between the words of a short sentence. Each row is a query word and each column a key word. Cell brightness is the softmax of the scaled dot product between projected embeddings. Sliders control the temperature, the vector dimension and a locality bias.

query ↓ key →thecatsleepsonthematthecatsleepsonthemat0.170.170.130.260.110.170.210.090.060.310.190.140.090.170.160.320.180.080.240.040.020.520.110.090.190.080.260.190.110.160.150.170.250.080.150.20
Mean row entropy (normalised)
0.92
Mean weight on itself
0.208
Causal mask
no
In a transformer, embeddings are projected to queries and keys and compared by a scaled dot product qi⊤kj/dq_i^{\top}k_j/\sqrt{d} before a softmax . The vectors here are random, not trained. Change the dimension and temperature to see how sharp the weights become.

Sources:7

QuestionTwo unit-length embeddings are at Euclidean distance 2\sqrt{2} from each other. What is their cosine similarity?

  1. A1
  2. B0
  3. C−1
  4. D2/2\sqrt{2}/2
Show the answer
  • A — Incorrect. A cosine of 1 means identical directions, hence distance 0.
  • B — Correct. From ∥u−v∥2=2−2cos⁡\lVert\mathbf{u}-\mathbf{v}\rVert^2 = 2 - 2\cos: 2=2−2cos⁡2 = 2 - 2\cos, so cos⁡=0\cos = 0. The vectors are orthogonal.
  • C — Incorrect. Opposite unit vectors are at distance 2, not 2\sqrt{2}.
  • D — Incorrect. That would give a squared distance of 2−2≈0.592 - \sqrt{2} \approx 0.59, i.e. a distance of about 0.77.

References

  1. Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean (2013). Efficient Estimation of Word Representations in Vector Space. arXiv preprint. arxiv.org/abs/1301.3781 unverified
  2. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean (2013). Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems 26 (NIPS 2013). unverified
  3. Omer Levy, Yoav Goldberg (2014). Neural Word Embedding as Implicit Matrix Factorization. Advances in Neural Information Processing Systems 27 (NIPS 2014). unverified
  4. Jeffrey Pennington, Richard Socher, Christopher D. Manning (2014). GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). unverified
  5. Tal Linzen (2016). Issues in evaluating semantic spaces using word analogies. Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP. unverified
  6. Kawin Ethayarajh (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP). unverified
  7. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017). arxiv.org/abs/1706.03762 unverified

Partager cette fiche

Mots-clés

Machine learningLarge language modelsEN

How Large Language Models Are Trained

From raw text to assistant: the published recipe behind large language models. Data preparation, next-token pretraining, scaling laws, supervised fine-tuning and preference tuning (RLHF, DPO), each with its objective written out.

Intermédiaire · 9 min de lecture ·

En vedetteIARecherche d’information

Le pipeline RAG, de l’indexation à la réponse

La génération augmentée par récupération donne à un modèle de langage des passages pertinents au moment de répondre. Sa qualité se joue d’abord à la récupération : ce qui n’est pas retrouvé ne peut pas être cité.

Intermédiaire · 11 min de lecture ·

En vedetteÉconométrieFinance

Le modèle GARCH(1,1) et la dynamique de la volatilité

Comment une récurrence à trois paramètres reproduit les grappes de volatilité des rendements financiers, à quelles conditions elle est stationnaire et comment on l’estime par quasi-maximum de vraisemblance.

Intermédiaire · 10 min de lecture ·