Machine learning · Explainer
How Large Language Models Are Trained
A large language model is a Transformer1 whose billions of parameters are tuned in two broad phases. Pretraining teaches it to predict the next token over a huge text corpus, and that is where almost all the compute goes. Post-training then reshapes it into an assistant with curated demonstrations and human preferences. This sheet follows that pipeline step by step, with the objective optimised at each stage.
The canonical training pipeline
- Data preparation
- Pretraining
- Post-training
- Human-provided signal
- Resulting model
- RLHF path (reward model)
- DPO path (no reward model)
- Prepare the data — Collect text, remove low-quality and duplicate documents, and convert everything into token IDs.
- Pretrain — Predict each next token over hundreds of billions to trillions of tokens. The result is a base model.
- Supervised fine-tuning — Continue training on written demonstrations of good answers.
- Preference tuning — Learn from human rankings, either through a reward model and RL (RLHF) or directly (DPO).
- Ship — Evaluate the tuned model and deploy it as an assistant.
Data
Step 1: Turn the internet into tokens
Collect
Large web crawls, books, code, encyclopaedic and scientific text. GPT-3 mixed a filtered Common Crawl with curated corpora3.
Filter
Heuristics and quality classifiers remove spam, boilerplate and low-quality pages. Many pipelines also try to strip toxic content and personal data.
Deduplicate
Removing exact and near-duplicate text reduces verbatim memorisation and can reach the same accuracy in fewer training steps7.
Tokenize
Text is split into subword units with byte-pair encoding or a variant6. The vocabulary typically has tens of thousands of entries, and each token becomes an integer ID.
Pretraining
Step 2: Pretraining — predict the next token
The model reads a sequence and outputs a probability for every vocabulary entry at each position. The loss is the average negative log-probability it gave to the token that actually came next. No labels are needed, because the text supplies its own targets (self-supervision). The gradient of is computed by backpropagation, and is updated with an Adam-type optimiser8 over many batches23.
| Symbol | Meaning | Unit |
|---|---|---|
| All trainable parameters of the Transformer (embeddings, attention and MLP weights) | ||
| Token observed at position t of the training sequence | ||
| All tokens before position t (causal context: the model cannot see ahead) | ||
| Probability the model assigns to the true next token, from a softmax over the vocabulary | ||
| Number of predicted tokens in the batch | tokens |
What pretraining adjusts: attention weights
Synthetic data — seed 7Attention weights of a single head. Embeddings (one vector per word plus a positional term) and the and projections are drawn from the seed: the map shows the mechanism, not what a trained model has learned. is a simplified relative-position bias; and give the standard formula.
Interactive heatmap of single-head attention weights between the words of a short sentence. Each row is a query word and each column a key word. Sliders change the softmax temperature, the vector dimension and a locality bias.
- Mean row entropy (normalised)
- 0.92
- Mean weight on itself
- 0.208
- Causal mask
- no
Each token costs about floating-point operations in the forward pass and about in the backward pass4.
| Symbol | Meaning | Unit |
|---|---|---|
| Total training compute | FLOPs | |
| Number of model parameters (non-embedding in Kaplan et al.) | ||
| Number of training tokens | tokens |
Sources:4
Scale is planned, not guessed. Test loss falls smoothly, roughly as a power law, as parameters, data and compute grow4. GPT-3 has billion parameters and was trained on billion tokens3, so FLOPs. Hoffmann et al. found that, for a fixed budget, parameters and tokens should grow together, at roughly 20 tokens per parameter. Their Chinchilla model (70 B parameters, 1.4 T tokens) outperformed the 4× larger Gopher trained with the same compute5.
Post-training
Step 3: Post-training — from base model to assistant
A base model continues text, but it does not reliably follow instructions. Supervised fine-tuning (SFT) keeps the same next-token loss but trains on pairs of a prompt and a human-written ideal response. The loss is often counted only on the response tokens. Preference tuning comes next. Labellers compare two answers, and a reward model learns to score the preferred one above the rejected one by minimising 911. The model is then optimised against that reward with PPO10. In InstructGPT, labellers preferred the outputs of a 1.3 B-parameter tuned model to those of the 175 B GPT-311.
Seek high reward without drifting too far from the reference model. Without the KL term, the policy could exploit flaws in the reward model. RLHF optimises this objective with reinforcement learning (PPO)11. DPO shows that the same objective can be optimised directly on preference pairs, with a simple classification-style loss and no separate reward model or RL loop12.
| Symbol | Meaning | Unit |
|---|---|---|
| The model being tuned (the policy), which generates response y to prompt x | ||
| Frozen reference model, usually the SFT model | ||
| Reward model score for response y to prompt x, learned from human rankings | ||
| Distribution of training prompts | ||
| Strength of the penalty for diverging from the reference model | ||
| Kullback–Leibler divergence between the tuned and reference output distributions |
| Criterion | Pretrainingmost compute | Supervised fine-tuning | Preference tuningRLHF / DPO |
|---|---|---|---|
| Data | Raw, unlabelled text | Prompts with human-written answers | Prompts with pairs of answers ranked by people |
| Objective | Next-token cross-entropy | Same loss, on demonstration responses | Reward under a KL penalty (PPO), or the DPO loss |
| Size | Hundreds of billions to trillions of tokens (GPT-3: 300 B; Chinchilla: 1.4 T) | Orders of magnitude smaller (tens of thousands of prompts in InstructGPT) | Similar order to SFT in InstructGPT |
| What changes | Language, facts and skills absorbed from the corpus | Format and instruction following | Which answers the model favours: helpfulness, tone, refusals |
Three stages, three kinds of data. Figures are from the cited papers.
Pretrainingmost compute
- Data
- Raw, unlabelled text
- Objective
- Next-token cross-entropy
- Size
- Hundreds of billions to trillions of tokens (GPT-3: 300 B; Chinchilla: 1.4 T)
- What changes
- Language, facts and skills absorbed from the corpus
Supervised fine-tuning
- Data
- Prompts with human-written answers
- Objective
- Same loss, on demonstration responses
- Size
- Orders of magnitude smaller (tens of thousands of prompts in InstructGPT)
- What changes
- Format and instruction following
Preference tuningRLHF / DPO
- Data
- Prompts with pairs of answers ranked by people
- Objective
- Reward under a KL penalty (PPO), or the DPO loss
- Size
- Similar order to SFT in InstructGPT
- What changes
- Which answers the model favours: helpfulness, tone, refusals
VerdictPretraining uses by far the most compute; InstructGPT's SFT and PPO stages cost a small fraction of GPT-3's pretraining. Post-training is cheap by comparison, yet it changes how the model behaves for users the most.
QuestionYou want to train a 10-billion-parameter model compute-optimally, in the Chinchilla sense. Roughly how many training tokens should you plan for?
- AAbout 10 billion (one token per parameter)
- BAbout 200 billion
- CAbout 10 trillion
- DIt doesn't matter, only the parameter count counts
Show the answer
- A — Incorrect. Far too few: the model would be heavily under-trained for its size.
- B — Correct. Right: about 20 tokens per parameter, so 20 × 10 B ≈ 200 B tokens. The budget is then C ≈ 6 × 10¹⁰ × 2×10¹¹ ≈ 1.2×10²² FLOPs.
- C — Incorrect. That is about 1,000 tokens per parameter. Training that long can still help, especially to make inference cheaper, but it is not the compute-optimal point.
- D — Incorrect. Scaling-law studies show that loss depends on both N and D. Neglecting data wastes compute.
References
- (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/1706.03762 unverified
- (2018). Improving Language Understanding by Generative Pre-Training. OpenAI technical report. unverified
- (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/2005.14165 unverified
- (2020). Scaling Laws for Neural Language Models. arXiv. arxiv.org/abs/2001.08361 unverified
- (2022). Training Compute-Optimal Large Language Models. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/2203.15556 unverified
- (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of ACL 2016. arxiv.org/abs/1508.07909 unverified
- (2022). Deduplicating Training Data Makes Language Models Better. Proceedings of ACL 2022. arxiv.org/abs/2107.06499 unverified
- (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR). arxiv.org/abs/1412.6980 unverified
- (2017). Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/1706.03741 unverified
- (2017). Proximal Policy Optimization Algorithms. arXiv. arxiv.org/abs/1707.06347 unverified
- (2022). Training Language Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/2203.02155 unverified
- (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems (NeurIPS). arxiv.org/abs/2305.18290 unverified