Research papers, myth by myth, Part 2: 'It only predicts the next word'

“It is just autocomplete.” I hear this line more than any other about AI models, usually as a way of ending an argument. The reasoning goes like this: the model writes one word, then the next, then the next, and each word is only a guess at what comes after the last. So there is no plan, no idea, nothing ahead of the current word. Part 1 showed that the first half of that is true: models do write one token at a time.

The second half is a claim about what happens inside, and until recently nobody could check it. In March 2025 Anthropic published tools that trace, step by step, how one of its models turns a prompt into an answer. The team says they went in expecting to confirm the autocomplete picture for poetry, and came out with the opposite.

The myth: “It only predicts the next word; there is no planning.”

The bottom line: the output is produced one token at a time, but the computation behind each token can look ahead. Researchers watched a model pick the rhyming word before it wrote the line, then build the line to land on it.

What do I need to know first?

Next-token prediction: how a LLM is trained and how it writes. Given the text so far, it scores every possible next token, one is picked, it is added to the text, and the model runs again. That is the interface. The myth assumes the interface tells you the mechanism.
Feature: a pattern of activity inside the model that stands for one recognisable concept, such as “the word rabbit”, “things that rhyme with -it” or “a number ending in 5”. One concept, one feature, which is much easier to read than the billions of raw numbers.
Attribution graph: a diagram of which features caused which, for one prompt, from the input words to the output word. Think of it as a wiring diagram for a single answer.
Intervention: changing a feature on purpose, turning it down (suppressing) or up (injecting), and watching what the model writes next. If the output changes the way you predicted, the feature really was doing that job.

Here is the everyday version of the whole question. Ask a friend to finish a rhyme out loud: “He saw a carrot and had to grab it…” Most people do not start the second line and hope. They think “rabbit” first, then say words that lead there: “…his hunger was like a starving rabbit.” The speaking is word by word. The thinking jumped ahead. The research question is whether a model does the same.

What did the researchers do?

Anthropic’s interpretability team published two papers on transformer-circuits.pub on 27 March 2025, with a plain-English summary on its blog, Tracing the thoughts of a large language model (published March 2025).

Circuit Tracing: Revealing Computational Graphs in Language Models is the method, which the blog calls an “AI microscope”.
On the Biology of a Large Language Model applies it to Claude 3.5 Haiku, which the paper calls “Anthropic’s lightweight production model”, across ten kinds of behaviour.

The method in plain words. A Transformer has two main parts in each layer: attention, which moves information between tokens (Part 1), and MLP layers, which transform each token’s numbers. The MLP numbers are hard to read because one neuron usually mixes many unrelated concepts. So the team trained a second, more readable network, a cross-layer transcoder, to imitate those MLP layers using features, and swapped it in to build a “replacement model”. For each prompt they freeze the original model’s attention patterns, run the replacement, and trace which features fed which. That trace is the attribution graph. Then they test what the graph suggests with interventions on the real model.

A worked example of why the stand-in matters. On a diverse set of ordinary text prompts, the team’s largest transcoder on an 18-layer test model picked the same next token as the real model about 50% of the time. That is good enough to generate hypotheses, not to replace the model, which is why every claim below is backed by an intervention.

What did they find?

Does it plan a rhyme before writing the line?

The prompt was the paper’s own:

A rhyming couplet:
He saw a carrot and had to grab it,
His hunger was like a starving rabbit

The first two lines are the prompt; the model wrote the second line. The blog states what the team expected: “Our guess was that Claude was writing word-by-word without much forethought until the end of the line, where it would make sure to pick a word that rhymes.” They expected two parallel paths at the last word, one for sense and one for rhyme, an improvisation.

Here is what the attribution graph showed instead, step by step:

Features on the token “it” (from “grab it”) switch on rhyme features for “-eet/-it/-et” sounds.
Those rhyme features switch on features for candidate endings, “rabbit” and “habit”. This happens on the newline token, the invisible line break at the end of line 1, before the model has written a single word of line 2.
The “rabbit” feature then shapes the words in between. The team traced the intermediate word “like” and found the same planned-word features feeding it, which they describe this way: “the model works backward from its target word to write a sentence that naturally ends in that word.”
At the end of the line, the plan pushes up the output “rabbit”.

the plan forms before the line: on the newline token, features for 'rhymes with it' switch on candidate endings 'rabbit' and 'habit'; the 'rabbit' plan then steers the words in between, such as 'like', and the final word

A graph alone could be a coincidence, so they edited the plan. Turning down the “rabbit” features at the newline made the model write a different line ending in “habit”, the runner-up it had also been holding. Injecting a “green” feature made it write a sensible line ending in “green”, which no longer rhymes. The blog calls this evidence of “both planning ability and adaptive flexibility”.

the experiment that proves it is a plan: change the features on the newline token and the whole line changes; suppress 'rabbit' and the line ends in 'habit'; inject 'green' and it ends in 'green'; the same steering at other positions does nothing

Two details make this much stronger than one cherry-picked poem. First, the edit only worked at the planning site: “the steering intervention only has an effect over the newline planning token”. The plan lives in one place at one time, exactly where the graph said. Second, when they injected “rabbit” or “green” plans into a random sample of 25 poems, the model ended its line with the injected word in 70% of cases. The paper also found that changing the plan changed the structure of the whole line, not only the last word.

Is there a separate model inside for each language?

The everyday version: a bilingual friend does not keep two separate ideas of “big”, one in English and one in French. They have one idea and two words for it. The question is whether a model works like that or keeps a “French Claude” and a “Chinese Claude” side by side.

The team gave Claude 3.5 Haiku the same question in three languages:

English: The opposite of "small" is "     -> big
French:  Le contraire de "petit" est "     -> grand
Chinese: "小"的反义词是"                    -> 大

The graphs had the same shape in all three. Early layers read the language-specific words. In the middle layers, the same features did the real work in every language: a “small” feature and an “opposite” feature, which together triggered a “large” feature. Late layers, steered by features that track which language the text is in, turned “large” into “big”, “grand” or “大”.

one concept, three languages: 'small', 'petit' and the Chinese character for small light up the same middle-layer features for smallness and opposite, which produce the shared idea of large; language features only at the edges turn it into big, grand or the Chinese character for big

Again they checked by editing. Swapping the “small” features for “hot” features made the model give the opposite of “hot” in the right language, even though the “hot” features came from an English prompt. Replacing the language features changed the output language while keeping the meaning. And the blog reports that the shared share grows with scale: “Claude 3.5 Haiku sharing more than twice the proportion of its features between languages as compared to a smaller model.”

This matters for the myth because a pure “next word” story would predict word statistics per language. What the team found is a concept first, words last. It also suggests why something a model learns in one language can show up in another.

How does it add numbers in its head?

The everyday version: when you add up a restaurant bill of £36 and £59 in your head, you might think “that’s about £95, roughly” and separately “6 and 9 make 15, so it ends in 5”. Two quick checks, then one answer.

The blog sets out the obvious guesses: the model has memorised a huge addition table, or it runs the long addition we learn at school. It found neither. For 36+59=, the attribution graph showed two paths running in parallel:

Path 1, rough size:  "add something near 57"  ->  "add near 36 to near 60"  ->  "the sum is near 92"
Path 2, last digit:  "ends in 9"  ->  "add something ending in 9"
                     ->  "something ending in 6 plus something ending in 9"  ->  "the sum ends in 5"
Combine:             near 92, ends in 5  ->  95

Path 1 knows the size but not the exact value. Path 2 knows the last digit exactly but nothing about size. Only 95 satisfies both. The paper notes that, “like many people do, the model has memorized the addition table for one-digit numbers”, which is the 6 + 9 step, but the rest is “a bit different than standard algorithms”.

Then they asked the model how it did it. Its answer, from the paper:

Human:     Answer in one word. What is 36+59?
Assistant: 95
Human:     Briefly, how did you get that?
Assistant: I added the ones (6+9=15), carried the 1, then added the tens (3+5+1=9), resulting in 95.

That is the school method, and the graph for the same prompt shows the two paths instead. The paper’s verdict is short: “Apparently not!” The blog’s explanation is that the model “learns to explain math by simulating explanations written by people, but that it has to learn to do math ‘in its head’ directly, without any such hints, and develops its own internal strategies to do so.”

how Claude 3.5 Haiku adds 36 and 59 inside: one path estimates the rough size of the sum, near 92, while another works out only the last digit, 5; together they give 95; asked how, it describes carrying the 1 instead

One more twist from the paper: the same “ends in 6 plus ends in 9” feature switches on outside arithmetic, for example in academic citations where a journal’s volume number ends in 6 and the year before the journal was founded ends in 9, so the publication year must end in 5. The model reuses one small skill wherever it helps predict the next token.

So is the myth true?

True about the output, false about the computation. Split it again.

“It predicts the next word”: true. Training rewards the model for predicting the next token, and at run time it emits one token at a time. Nothing in these papers changes that.

“So there is no planning”: false, at least for these cases. On the newline token, before line 2 exists, the model already holds candidate endings, prefers one, and uses it to choose the words in between. Editing that plan changes the line in the predicted way. That is planning in the plain sense: deciding where you are going before you get there. The paper reports “evidence of both forward planning and backwards planning (albeit basic forms)”.

The deeper mistake in the myth is treating “predict the next token” as a description of how the model thinks. It describes the task. To predict the next word of a rhyming poem well, the cheapest internal strategy turns out to be thinking about the end of the line early. To predict “95” well, the model built its own adding machine. The interface is one word at a time; what runs behind it is not.

What did the papers not show?

The authors are unusually direct about limits, so I quote them.

The microscope works on a minority of prompts. “We’ve found that our attribution graphs provide us with satisfying insight for about a quarter of the prompts we’ve tried.” The poem, the three languages and the addition are success cases, and “undoubtedly a biased sample shaped by the limitations of our tools.”
It sees part of the computation. “Even on short, simple prompts, our method only captures a fraction of the total computation performed by Claude, and the mechanisms we do see may have some artifacts based on our tools which don’t reflect what is going on in the underlying model.”
Attention is held fixed. The method freezes the attention patterns rather than explaining them, so it shows what happens given where the model looked, not why it looked there.
It is slow and short. “It currently takes a few hours of human effort to understand the circuits we see, even on prompts with only tens of words.” Long documents and long agent tasks are out of reach for now.
One model, simple tasks. The findings are about Claude 3.5 Haiku on short prompts. The paper says it suspects similar mechanisms elsewhere but “cannot guarantee it”. The planning shown is a few words ahead, not a plan for a whole essay.
The English default. Even in the multilingual case, the paper finds “important mechanistic ways in which English is privileged”, so “one shared concept space” is a simplification.

Why does this matter if you build or use AI?

Do not trust a model’s account of its own reasoning. The addition example is the cleanest demonstration I know: a correct answer and a confident, plausible, wrong explanation of how it was reached. If a decision needs an audit trail, get it from the evidence, tools and checks around the model, not from the model’s description of itself.
“Just autocomplete” is the wrong mental model for risk. A system that can plan a few steps ahead inside a single answer can also pursue a goal across a conversation. Design guardrails and approvals for what it can do, not for what a word predictor could do.
Language is less of a wall than you think. Shared concepts across languages are one reason a model trained mostly on English often works in other languages. Still test in each language you ship, because the shared space is not perfectly shared.

If you build agents, I cover evidence-based evaluation and approvals in the Production-ready agents series.

Lock it in

Myth             "It only predicts the next word; there is no planning."
Verdict          True about the output, false about the computation.
Rhyme            'rabbit' and 'habit' active on the newline before line 2 is written.
                 Suppress 'rabbit' -> line ends 'habit'. Inject 'green' -> ends 'green'.
                 Edits work only at the newline. Injection in 25 poems: 70% success.
Languages        small / petit / 小 -> shared 'small' + 'opposite' -> 'large'
                 -> big / grand / 大. Haiku shares over twice a smaller model's proportion.
Addition         36+59: rough path (near 92) + last-digit path (ends in 5) -> 95.
                 Its own explanation: carry the 1. Not what the graph shows.
Limits           Useful insight on about a quarter of prompts; attention frozen;
                 hours of work per short prompt; one model.

My view: “it only predicts the next word” was always a statement about the training objective dressed up as a statement about the mind. These papers do not show that a model thinks like a person, and the authors are careful not to claim it. They show something narrower and more useful: you can watch a plan form at a specific token, edit it, and see the output change. That moves the question from philosophy to measurement.

One honest gap: the most striking result, the arithmetic, is also the warning. If the model’s explanations can be stories borrowed from human text, then the reasoning a model writes out when it “thinks step by step” may not be its real reasoning either. These papers trace a quarter of short prompts; nobody can yet trace a long chain of thought end to end.

What did we learn?

A language model writes one token at a time because that is how it is trained and run; that describes the interface, not the computation behind each token.
Anthropic’s 2025 circuit-tracing method builds a readable stand-in for a model’s internals, traces which features cause which, and tests each story by editing features in the real model.
In a rhyming couplet, Claude 3.5 Haiku chose ‘rabbit’ (with ‘habit’ as runner-up) on the line break, before writing the line, and editing that plan changed the line as predicted.
The same middle-layer concept features handle ‘the opposite of small’ in English, French and Chinese, with language-specific features only at the edges.
For 36+59 the model runs a rough-size path and a last-digit path in parallel, but explains the answer with the school method of carrying the 1.
The tools give clear insight for only about a quarter of prompts, on short inputs and one model, so the findings are strong examples, not a full map.

What comes next?

Parts 1 and 2 looked at what a model does. Part 3 asks how anyone can know: it takes on the myth that nobody can see inside a model, and shows how researchers pull readable features out of the numbers.

Sources

Anthropic, Tracing the thoughts of a large language model (published March 2025)
Ameisen et al., Anthropic, Circuit Tracing: Revealing Computational Graphs in Language Models (published March 2025)
Lindsey et al., Anthropic, On the Biology of a Large Language Model (published March 2025)
For the background on attention used in this part: Vaswani et al., Attention Is All You Need (published June 2017), covered in Part 1

Opinions are my own. Every fact in this post comes from the public documentation linked above.