Aller au contenu

Machine learningLarge language modelsNatural language processingExplicationIntermédiaire

Par · Publiée le · 9 min de lecture

Sommaire
  1. Step 1: Turn the internet into tokens
  2. Step 2: Pretraining — predict the next token
  3. Step 3: Post-training — from base model to assistant

Après cette fiche, vous saurez

  • Name the main stages of LLM training and what each one produces
  • Write the next-token cross-entropy loss and define every term
  • Estimate training compute with C ≈ 6ND and apply the compute-optimal tokens-per-parameter rule of thumb
  • Explain how supervised fine-tuning, reward models, RLHF and DPO turn a base model into an assistant

Formulaire

Les équations de la fiche, d’un coup d’œil

  1. 01Next-token cross-entropyL(θ)=−1T∑t=1Tlog⁡pθ(xt∣x<t)\mathcal{L}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta\left(x_t \mid x_{<t}\right)
  2. 02Training computeC≈6 N DC \approx 6\,N\,D
  3. 03KL-regularised preference objectivemax⁡πθ  Ex∼D,  y∼πθ(⋅∣x)[rϕ(x,y)]  −  β DKL[πθ(y∣x) ∥ πref(y∣x)]\max_{\pi_\theta}\; \mathbb{E}_{x\sim\mathcal{D},\; y\sim\pi_\theta(\cdot\mid x)}\big[r_\phi(x,y)\big] \; - \; \beta\, \mathbb{D}_{\mathrm{KL}}\big[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\big]

Machine learning · Explainer

How Large Language Models Are Trained

A large language model is a Transformer1 whose billions of parameters are tuned in two broad phases. Pretraining teaches it to predict the next token over a huge text corpus, and that is where almost all the compute goes. Post-training then reshapes it into an assistant with curated demonstrations and human preferences. This sheet follows that pipeline step by step, with the objective optimised at each stage.

The canonical training pipeline

Flow diagram in three groups. Data preparation: raw text is filtered and deduplicated, then tokenized. Pretraining: token sequences feed next-token pretraining, which produces a base model. Post-training: the base model initialises supervised fine-tuning. Human preference data trains a reward model, which gives a reward signal to the RL step (RLHF path). Alternatively, preference data feeds the policy directly (DPO path). The tuned policy becomes the assistant model.DATA PREPARATIONPRETRAININGPOST-TRAININGRaw textweb, books, code, articlesFilter & deduplicatequality, safety,near-duplicatesTokenizesubword units (BPE)Pretrainingpredict the next tokenBase modelcontinues text, does not followinstructions reliablySupervised fine-tuningprompt → ideal responsePreference datahumans rank pairs of answersReward modelscores a responsePolicy optimisationPPO with KL penalty, or DPOAssistant modelevaluated, then deployedcleansplit into tokenstoken sequenceslearned weightsinitialisestarting policytrain on rankingsreward signalused directly (DPO)tuned weights
  • Data preparation
  • Pretraining
  • Post-training
  • Human-provided signal
  • Resulting model
  • RLHF path (reward model)
  • DPO path (no reward model)
  1. Prepare the data — Collect text, remove low-quality and duplicate documents, and convert everything into token IDs.
  2. Pretrain — Predict each next token over hundreds of billions to trillions of tokens. The result is a base model.
  3. Supervised fine-tuning — Continue training on written demonstrations of good answers.
  4. Preference tuning — Learn from human rankings, either through a reward model and RL (RLHF) or directly (DPO).
  5. Ship — Evaluate the tuned model and deploy it as an assistant.
The published recipe (GPT-3, InstructGPT, DPO). Frontier labs add their own variants, which are often undisclosed.

Data

Step 1: Turn the internet into tokens

  • Collect

    Large web crawls, books, code, encyclopaedic and scientific text. GPT-3 mixed a filtered Common Crawl with curated corpora3.

  • Filter

    Heuristics and quality classifiers remove spam, boilerplate and low-quality pages. Many pipelines also try to strip toxic content and personal data.

  • Deduplicate

    Removing exact and near-duplicate text reduces verbatim memorisation and can reach the same accuracy in fewer training steps7.

  • Tokenize

    Text is split into subword units with byte-pair encoding or a variant6. The vocabulary typically has tens of thousands of entries, and each token becomes an integer ID.

Pretraining

Step 2: Pretraining — predict the next token

Next-token cross-entropy
L(θ)=−1T∑t=1Tlog⁡pθ(xt∣x<t)\mathcal{L}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta\left(x_t \mid x_{<t}\right)

The model reads a sequence and outputs a probability for every vocabulary entry at each position. The loss is the average negative log-probability it gave to the token that actually came next. No labels are needed, because the text supplies its own targets (self-supervision). The gradient of L\mathcal{L} is computed by backpropagation, and θ\theta is updated with an Adam-type optimiser8 over many batches23.

Variables
SymbolMeaningUnit
θ\thetaAll trainable parameters of the Transformer (embeddings, attention and MLP weights)
xtx_tToken observed at position t of the training sequence
x<tx_{<t}All tokens before position t (causal context: the model cannot see ahead)
pθ(xt∣x<t)p_\theta(x_t \mid x_{<t})Probability the model assigns to the true next token, from a softmax over the vocabulary
TTNumber of predicted tokens in the batchtokens

Sources:23

What pretraining adjusts: attention weights

Synthetic data — seed 7

Attention weights softmax⁡((qi⊤kj/d−λ ∣i−j∣)/τ)\operatorname{softmax}\big((q_i^{\top} k_j/\sqrt{d} - \lambda\,|i-j|)/\tau\big) of a single head. Embeddings (one vector per word plus a positional term) and the QQ and KK projections are drawn from the seed: the map shows the mechanism, not what a trained model has learned. λ\lambda is a simplified relative-position bias; τ=1\tau = 1 and λ=0\lambda = 0 give the standard formula.

Interactive heatmap of single-head attention weights between the words of a short sentence. Each row is a query word and each column a key word. Sliders change the softmax temperature, the vector dimension and a locality bias.

query ↓ key →thecatsleepsonthematthecatsleepsonthemat0.170.170.130.260.110.170.210.090.060.310.190.140.090.170.160.320.180.080.240.040.020.520.110.090.190.080.260.190.110.160.150.170.250.080.150.20
Mean row entropy (normalised)
0.92
Mean weight on itself
0.208
Causal mask
no
The Q and K projections here are random, as they are at initialisation. Pretraining changes these projections, among other weights, until the attention patterns help predict the next token. Vary the temperature to see the softmax sharpen or flatten.
Training compute
C≈6 N DC \approx 6\,N\,D

Each token costs about 2N2N floating-point operations in the forward pass and about 4N4N in the backward pass4.

Variables
SymbolMeaningUnit
CCTotal training computeFLOPs
NNNumber of model parameters (non-embedding in Kaplan et al.)
DDNumber of training tokenstokens

Sources:4

Scale is planned, not guessed. Test loss falls smoothly, roughly as a power law, as parameters, data and compute grow4. GPT-3 has N≈175N \approx 175 billion parameters and was trained on D≈300D \approx 300 billion tokens3, so 6ND≈3.2×10236ND \approx 3.2\times10^{23} FLOPs. Hoffmann et al. found that, for a fixed budget, parameters and tokens should grow together, at roughly 20 tokens per parameter. Their Chinchilla model (70 B parameters, 1.4 T tokens) outperformed the 4× larger Gopher trained with the same compute5.

Sources:435

Post-training

Step 3: Post-training — from base model to assistant

A base model continues text, but it does not reliably follow instructions. Supervised fine-tuning (SFT) keeps the same next-token loss but trains on pairs of a prompt and a human-written ideal response. The loss is often counted only on the response tokens. Preference tuning comes next. Labellers compare two answers, and a reward model rϕr_\phi learns to score the preferred one ywy_w above the rejected one yly_l by minimising −log⁡σ(rϕ(x,yw)−rϕ(x,yl))-\log\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big)911. The model is then optimised against that reward with PPO10. In InstructGPT, labellers preferred the outputs of a 1.3 B-parameter tuned model to those of the 175 B GPT-311.

Sources:91110

KL-regularised preference objective
max⁡πθ  Ex∼D,  y∼πθ(⋅∣x)[rϕ(x,y)]  −  β DKL[πθ(y∣x) ∥ πref(y∣x)]\max_{\pi_\theta}\; \mathbb{E}_{x\sim\mathcal{D},\; y\sim\pi_\theta(\cdot\mid x)}\big[r_\phi(x,y)\big] \; - \; \beta\, \mathbb{D}_{\mathrm{KL}}\big[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\big]

Seek high reward without drifting too far from the reference model. Without the KL term, the policy could exploit flaws in the reward model. RLHF optimises this objective with reinforcement learning (PPO)11. DPO shows that the same objective can be optimised directly on preference pairs, with a simple classification-style loss and no separate reward model or RL loop12.

Variables
SymbolMeaningUnit
πθ\pi_\thetaThe model being tuned (the policy), which generates response y to prompt x
πref\pi_{\mathrm{ref}}Frozen reference model, usually the SFT model
rϕ(x,y)r_\phi(x,y)Reward model score for response y to prompt x, learned from human rankings
D\mathcal{D}Distribution of training prompts
β\betaStrength of the penalty for diverging from the reference model
DKL\mathbb{D}_{\mathrm{KL}}Kullback–Leibler divergence between the tuned and reference output distributions

Sources:1112

Three stages, three kinds of data. Figures are from the cited papers.
CriterionPretrainingmost computeSupervised fine-tuningPreference tuningRLHF / DPO
DataRaw, unlabelled textPrompts with human-written answersPrompts with pairs of answers ranked by people
ObjectiveNext-token cross-entropySame loss, on demonstration responsesReward under a KL penalty (PPO), or the DPO loss
SizeHundreds of billions to trillions of tokens (GPT-3: 300 B; Chinchilla: 1.4 T)Orders of magnitude smaller (tens of thousands of prompts in InstructGPT)Similar order to SFT in InstructGPT
What changesLanguage, facts and skills absorbed from the corpusFormat and instruction followingWhich answers the model favours: helpfulness, tone, refusals

Three stages, three kinds of data. Figures are from the cited papers.

Pretrainingmost compute

Data
Raw, unlabelled text
Objective
Next-token cross-entropy
Size
Hundreds of billions to trillions of tokens (GPT-3: 300 B; Chinchilla: 1.4 T)
What changes
Language, facts and skills absorbed from the corpus

Supervised fine-tuning

Data
Prompts with human-written answers
Objective
Same loss, on demonstration responses
Size
Orders of magnitude smaller (tens of thousands of prompts in InstructGPT)
What changes
Format and instruction following

Preference tuningRLHF / DPO

Data
Prompts with pairs of answers ranked by people
Objective
Reward under a KL penalty (PPO), or the DPO loss
Size
Similar order to SFT in InstructGPT
What changes
Which answers the model favours: helpfulness, tone, refusals

VerdictPretraining uses by far the most compute; InstructGPT's SFT and PPO stages cost a small fraction of GPT-3's pretraining. Post-training is cheap by comparison, yet it changes how the model behaves for users the most.

Sources:3511

QuestionYou want to train a 10-billion-parameter model compute-optimally, in the Chinchilla sense. Roughly how many training tokens should you plan for?

  1. AAbout 10 billion (one token per parameter)
  2. BAbout 200 billion
  3. CAbout 10 trillion
  4. DIt doesn't matter, only the parameter count counts
Show the answer
  • A — Incorrect. Far too few: the model would be heavily under-trained for its size.
  • B — Correct. Right: about 20 tokens per parameter, so 20 × 10 B ≈ 200 B tokens. The budget is then C ≈ 6 × 10¹⁰ × 2×10¹¹ ≈ 1.2×10²² FLOPs.
  • C — Incorrect. That is about 1,000 tokens per parameter. Training that long can still help, especially to make inference cheaper, but it is not the compute-optimal point.
  • D — Incorrect. Scaling-law studies show that loss depends on both N and D. Neglecting data wastes compute.

References

  1. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/1706.03762 unverified
  2. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever (2018). Improving Language Understanding by Generative Pre-Training. OpenAI technical report. unverified
  3. Tom B. Brown, et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/2005.14165 unverified
  4. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei (2020). Scaling Laws for Neural Language Models. arXiv. arxiv.org/abs/2001.08361 unverified
  5. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. (2022). Training Compute-Optimal Large Language Models. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/2203.15556 unverified
  6. Rico Sennrich, Barry Haddow, Alexandra Birch (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of ACL 2016. arxiv.org/abs/1508.07909 unverified
  7. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini (2022). Deduplicating Training Data Makes Language Models Better. Proceedings of ACL 2022. arxiv.org/abs/2107.06499 unverified
  8. Diederik P. Kingma, Jimmy Ba (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR). arxiv.org/abs/1412.6980 unverified
  9. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei (2017). Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/1706.03741 unverified
  10. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov (2017). Proximal Policy Optimization Algorithms. arXiv. arxiv.org/abs/1707.06347 unverified
  11. Long Ouyang, Jeff Wu, Xu Jiang, et al. (2022). Training Language Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/2203.02155 unverified
  12. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/2305.18290 unverified

Partager cette fiche

Mots-clés

Machine learningNatural language processingEN

The Mathematics of Embeddings

How discrete symbols become vectors: the embedding lookup as a matrix product, the similarity measures used to compare vectors, how skip-gram training relates to a factorised co-occurrence matrix, and what high-dimensional geometry implies for reading similarity scores.

Intermédiaire · 10 min de lecture ·

En vedetteIARecherche d’information

Le pipeline RAG, de l’indexation à la réponse

La génération augmentée par récupération donne à un modèle de langage des passages pertinents au moment de répondre. Sa qualité se joue d’abord à la récupération : ce qui n’est pas retrouvé ne peut pas être cité.

Intermédiaire · 11 min de lecture ·

En vedetteÉconométrieFinance

Le modèle GARCH(1,1) et la dynamique de la volatilité

Comment une récurrence à trois paramètres reproduit les grappes de volatilité des rendements financiers, à quelles conditions elle est stationnaire et comment on l’estime par quasi-maximum de vraisemblance.

Intermédiaire · 10 min de lecture ·