In-Context Learning and Induction Heads — Anthropic 2022
Large language models can pick up a pattern from earlier in a prompt and continue it — no weight updates, just context. This is "in-context learning," and one of the most beautiful results in mechanistic interpretability is that we can trace much of it back to a specific, nameable circuit: the induction head.
This video walks through Anthropic's "In-context Learning and Induction Heads" (Olsson et al., 2022). We start with a mysterious bump in the training loss curve, define and measure in-context learning, then meet the induction head — a two-head circuit that implements the rule [A][B]...[A] → [B]: find where the current token appeared before, and predict what came next.
Then we go through the six lines of evidence the authors assemble to argue that induction heads are the main mechanism behind in-context learning in transformers:
• Argument 1 — Co-occurrence: induction heads and in-context learning appear together, in the same phase change.
• Argument 2 — Co-perturbation: shift when the heads can form, and the learning ability shifts with them.
• Argument 3 — Ablation: knock the heads out, and in-context learning drops.
• Argument 4 — Generality: the heads do more than literal copying — translation, pattern completion.
• Argument 5 — Mechanism: in small models we can literally read the circuit off the weights.
• Argument 6 — Continuity: the behavior is smooth from small to large models.
A clear, visual tour of how a sudden capability emerges during training — and why that matters for AI safety.
⏱️ Chapters
0:00 A Bump in the Loss Curve
0:48 What Is In-Context Learning?
1:35 Putting a Number on It
2:20 Meet the Induction Head
3:05 It Takes Two Heads
3:49 The Phase Change
4:33 Seeing It Token by Token
5:18 Tracking the Heads Form
6:04 Building the Case
6:42 Argument 1: They Appear Together
7:24 Argument 2: Move One, Move the Other
8:07 Argument 3: Take Them Away
8:49 Argument 4: More Than Copying
9:36 Argument 5: We Can Read the Mechanism
10:21 Argument 6: Smooth From Small to Large
11:14 Why It Matters
📄 Source
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, et al., "In-context Learning and Induction Heads," Transformer Circuits Thread (2022). https://transformer-circuits.pub/2022...
Builds on "A Mathematical Framework for Transformer Circuits" (2021). All credit to the Anthropic interpretability team; figures and slates here are for educational explanation.
We're brand new to YouTube — if this helped, please like and subscribe. It genuinely keeps these explainers coming. 🙏
#InductionHeads #InContextLearning #MechanisticInterpretability #Transformers #LLM #AI #MachineLearning #AISafety #Anthropic #NeuralNetworks