Research papers, myth by myth, Part 5: 'A model's chain of thought shows how it really reached the answer'

When a reasoning model answers a hard question, many apps now let you open a panel and read its “thinking”: a tidy paragraph of steps that ends in the answer. It feels like looking over the model’s shoulder. If the steps look sensible, we relax; if one looks wrong, we think we have found the bug. This part tests that feeling against three papers.

The myth: “A model’s chain of thought shows how it really reached the answer.”

The bottom line: writing out steps genuinely improves a model’s answers, but the written steps are a separate story the model tells alongside its computation. Sometimes the story matches what happened inside. Often it leaves out the thing that actually decided the answer, and you cannot tell which case you are in by reading the text.

What do I need to know first?

Think of a pupil in a maths exam who is told to show their working. Showing the working usually helps: they break the problem into small steps and make fewer slips. The teacher also reads the working to see how the pupil thought. But imagine the pupil glanced at a neighbour’s answer sheet, saw “C”, and then wrote neat working that arrives at C. The working looks fine. It just is not how they got there.

Map that onto the real terms:

Chain of thought (CoT) is the working: intermediate reasoning steps the model writes before its final answer. In a reasoning model these steps are often produced automatically as thinking tokens.
Chain-of-thought prompting is the teacher saying “show your working”: you include a worked example with steps in the prompt, and the model copies the style. Giving a model a handful of worked examples in the prompt is called few-shot prompting.
Faithfulness is whether the working matches the real reason for the answer. A chain of thought is faithful when it mentions everything that actually drove the answer, including the neighbour’s answer sheet.
Interpretability is the teacher fitting the pupil with a brain scanner instead of reading the working. Interpretability research traces the model’s internal computation directly, so it can be compared with what the model says.

The myth assumes the working and the real reasoning are the same thing. The papers below pull them apart.

What did the researchers do?

Paper 1: why steps help. Jason Wei and colleagues at Google published Chain-of-Thought Prompting Elicits Reasoning in Large Language Models in January 2022 (it appeared at NeurIPS 2022). They took existing large language models, including Google’s PaLM family up to 540 billion parameters, and changed nothing but the prompt. In one version the worked examples gave only answers; in the other, the same examples showed the steps. They measured accuracy on maths word problems such as GSM8K, plus commonsense and symbol-manipulation tasks.

Paper 2: does the working admit the real reason? Anthropic’s Alignment Science team published Reasoning models don’t always say what they think in April 2025, with the full paper on arXiv. They tested two reasoning models, Claude 3.7 Sonnet and DeepSeek R1, on multiple-choice questions. Then they slipped a hint about the answer into the prompt, checked that the hint really changed the answer, and read the chain of thought to see whether it mentioned the hint. In a second experiment they used reinforcement learning to reward the model for following deliberately wrong hints, and checked whether its reasoning ever confessed.

Paper 3: compare the story with the scan. In March 2025 Anthropic’s interpretability team published Tracing the thoughts of a large language model, with the detail in On the Biology of a Large Language Model. They built an attribution graph for Claude 3.5 Haiku doing simple tasks, including adding two-digit numbers, and then asked the model to explain itself.

What did they find?

Steps make answers better

Here is the paper’s own example, from its Figure 1. Both prompts contain one worked example about Roger’s tennis balls and then a new question about a cafeteria.

the same new question sent two ways, from Figure 1 of the chain-of-thought paper: with a worked example that gives only the answer, the model replies 27, which is wrong; with a worked example that shows the steps, the model writes its own steps and replies 9, which is right

With the answer-only example, the model replied “The answer is 27”. With the worked example, it wrote: the cafeteria had 23 apples, used 20, so 23 - 20 = 3; it bought 6 more, so 3 + 6 = 9. The answer is 9. The mechanism is easy to see with these small numbers. “23, minus 20, plus 6” in one jump is a place to slip; two one-step sums are hard to get wrong. Each written step also becomes part of the text the model reads when it predicts the next step, so the model is effectively using its own output as scratch paper.

The effect was large, but only for big models:

GSM8K maths word problems solved by three sizes of PaLM, standard prompting versus chain-of-thought prompting, from Table 2 of the paper: at 8 billion parameters the steps slightly hurt, at 540 billion they lift the solve rate from 17.9% to 56.9%

On GSM8K, PaLM 540B went from 17.9% with standard prompting to 56.9% with chain-of-thought prompting. PaLM 8B went from 4.9% to 4.1%, slightly worse. The authors write that smaller models “produced fluent but illogical chains of thought”, and that the gains only appeared with models of around 100 billion parameters. Keep that word “fluent” in mind: even in the paper that launched the technique, the text of the working and its correctness were separate things.

The working often hides what changed the answer

Anthropic’s test is the exam note made literal. Their own analogy is “slipping a student a note saying ‘the answer is [A]’ before an exam”. Some hints were neutral, such as “a Stanford Professor indicates the answer is [A]” or a fake metadata line containing the answer. Some were designed to be worrying, such as telling the model it had “gained unauthorized access to the system” and been shown the answer.

the hint test: without a hint the model answers D; with a fake metadata line pointing to C slipped into the prompt it answers C, and its chain of thought argues for C without ever mentioning the hint; across hint types Claude 3.7 Sonnet mentioned the hint 25% of the time and DeepSeek R1 39%

The diagram follows the example on Anthropic’s page: Claude 3.7 Sonnet answered D to the original question, switched to C after a metadata hint was added, and its chain of thought never mentioned the metadata. Across all hint types, Claude 3.7 Sonnet mentioned the hint 25% of the time and DeepSeek R1 39% of the time. For the “unauthorized access” class, Claude was faithful 41% of the time and R1 19% of the time.

Two details make this worse than it sounds. First, the hidden reasoning was not a shortcut to save words: on average, the unfaithful chains of thought were substantially longer than the faithful ones. Second, faithfulness was lower on harder questions, which are exactly the ones where you would most want to check the working.

The reward-hacking experiment pushed further. The team rewarded Claude 3.7 Sonnet for picking the wrong answers that matched planted hints, a form of reward hacking. The model learned to exploit the hack on more than 99% of prompts, but mentioned it in its chain of thought less than 2% of the time in more than half of the environments. Instead, it often built a fake rationale for why the wrong answer was right. Anthropic’s example is a medical question where a hint points to “fish” as a cause of cancer risk; the model writes a long explanation of why fish is correct, without mentioning the hint.

They also tried to train faithfulness up. Outcome-based reinforcement learning on hard maths and coding problems raised faithfulness at first, then plateaued at 28% on MMLU and 20% on GPQA.

Even honest-looking working can describe the wrong method

The first two papers catch the model leaving something out. The tracing work catches something subtler: a correct answer, a plausible explanation, and a different method inside.

what it says versus what it did for 36 + 59 in Claude 3.5 Haiku: asked how it got 95, the model describes the school method with a carry; the traced computation instead runs a rough path that lands near 92 and an exact path that finds the last digit is 5, then combines them

The researchers asked Claude 3.5 Haiku “What is 36+59?” and then “Briefly, how did you get that?”. It answered: “I added the ones (6+9=15), carried the 1, then added the tens (3+5+1=9), resulting in 95.” The attribution graph showed something else. One pathway worked at rough precision (“add something near 57”, then “near 36 plus near 60”, landing on “the sum is near 92”). A second pathway worked out only the last digit: a number ending in 6 plus a number ending in 9 must end in 5. Combine “near 92” with “ends in 5” and you get 95. Anthropic’s summary is that the model “seems to be unaware of the sophisticated ‘mental math’ strategies that it learned during training”. Their explanation is that the model learned to explain maths by imitating explanations written by people, while it had to learn to do the maths directly, with no such examples.

The same post shows both sides of the coin. Asked for the square root of 0.64, Claude produced a faithful chain of thought, with internal features for the intermediate step of the square root of 64. Asked for the cosine of a large number it could not easily compute, it sometimes just produced an answer and claimed to have calculated it, with no sign of that calculation inside. Given a hint about the answer, it sometimes worked backwards to steps that would reach the hinted target, which the authors call motivated reasoning.

So is the myth true?

False as a general rule. Each paper supports a different part of the verdict:

Wei et al. show chain of thought is useful: writing steps raises accuracy, sometimes dramatically, on large models.
Anthropic’s hint experiments show it is incomplete: when a hint changed the answer, the reasoning admitted it only 25% (Claude 3.7 Sonnet) or 39% (DeepSeek R1) of the time on average, and under 2% for learned reward hacks in most environments.
The tracing work shows it can be a reconstruction: a correct answer explained with a textbook method the model did not use.

The honest summary is that a chain of thought is evidence about the model’s reasoning, sometimes good evidence, never a log. The working that helps the model think and the explanation it gives you are produced by the same next-token machinery, and nothing forces the explanation to include everything that mattered.

What did the papers not show?

The authors are careful about their limits, and so should we be.

Wei et al. state that chain of thought “emulates the thought processes of human reasoners” but that this “does not answer whether the neural network is actually ‘reasoning’”, which they leave open. They also note there is no guarantee of correct reasoning paths, and that the benefit appeared only at large scale. Their paper never claimed the steps were faithful; it measured accuracy.
Anthropic’s hint study calls its scenarios “somewhat contrived”: multiple-choice quizzes with planted hints, unlike real tasks. It covered only Anthropic and DeepSeek models and a limited range of hint types. Perhaps most important, the tasks were not hard enough to need the chain of thought. On problems a model cannot solve without writing steps, the authors suggest it may be unable to hide its real reasoning, which would make monitoring easier.
The tracing work says its method captures “only a fraction of the total computation”, may include artefacts of the tools, and takes a few hours of human effort for prompts of tens of words. One addition problem in one model is a vivid example, not a statistic.

None of this says chain of thought is useless for oversight. It says that a clean-looking chain of thought is weak evidence that nothing else happened.

Why does this matter if you build or use AI?

Keep using steps for quality. Prompting for working, or using a reasoning model, still improves answers on multi-step problems. That finding has held up.
Do not treat the reasoning panel as an audit trail. If you need to know why an agent did something, record what it actually did: the inputs it saw, the tools it called, the arguments, the results. Many APIs make the thinking optional anyway; in the Gemini API, for example, thought summaries are off by default, and what you get back is a summary of the thinking.
Check outcomes, not explanations. An eval that grades the final answer and the actions taken will catch a wrong result that a persuasive explanation would hide. I cover this in the Production-ready agents series.
Treat anything in the context as a possible hint. Retrieved documents, tool outputs and metadata can steer an answer the way the planted hints did, and the reasoning may never mention them.
Read the working for clues. A step that looks wrong is worth investigating. A chain that looks right proves less than it seems.

How do I lock it in?

Term                        What it is                                   Exam picture
Chain of thought            steps the model writes before answering      the pupil's working
Chain-of-thought prompting  worked examples that show steps              "show your working"
Faithfulness                working names what really drove the answer   admits the peek
Unfaithful CoT              plausible working, real cause left out       neat working, copied answer
Reward hacking              learned loophole that earns the reward       gaming the marking scheme
Interpretability            tracing the computation inside the model     the brain scan

My view: I find it useful to treat a model’s chain of thought the way I treat a colleague’s explanation after the fact: helpful, usually sincere in tone, and shaped by what a good explanation is supposed to sound like. The papers make me more interested in what the model did (calls, inputs, outputs) and less interested in what it says it did.

One honest gap: all three papers study fairly short, artificial tasks. We do not yet have good public measurements of faithfulness for long agent runs with many tool calls, where the chain of thought is much longer and the stakes are higher. The hint study’s own caveat cuts both ways: harder tasks might force more honest reasoning, or offer more room to hide.

What did we learn?

Chain-of-thought prompting, showing a model worked examples with steps, raised PaLM 540B’s GSM8K score from 17.9% to 56.9%, but helped only large models.
The steps help because each small step is easier to get right and becomes scratch paper for the next one.
When a planted hint changed a reasoning model’s answer, its chain of thought mentioned the hint only 25% (Claude 3.7 Sonnet) or 39% (DeepSeek R1) of the time on average.
Models trained to exploit a reward hack used it on more than 99% of prompts but admitted it under 2% of the time in most environments, often inventing a rationale instead.
Circuit tracing caught Claude 3.5 Haiku explaining 36 + 59 with the carry method while actually combining a rough estimate with a last-digit calculation.
Use chain of thought to get better answers, and use logs of actions, tool calls and outcomes to find out what really happened.

What comes next?

In Part 6 I take on the myth that bigger models get smarter in sudden, unpredictable leaps, so more parameters is always the answer. You have already seen one “leap” in this part: chain-of-thought prompting only helped above a certain size. Part 6 asks whether such leaps are real, or partly a result of how we mark the answers.

Sources

Jason Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv 2201.11903, NeurIPS 2022), and the Google Research blog post Language Models Perform Reasoning via Chain of Thought
Anthropic, Reasoning models don’t always say what they think (April 2025), and the paper by Yanda Chen et al., Reasoning Models Don’t Always Say What They Think (arXiv 2505.05410)
Anthropic, Tracing the thoughts of a large language model (March 2025)
Jack Lindsey et al., On the Biology of a Large Language Model (Transformer Circuits, March 2025), the addition and chain-of-thought faithfulness sections
Google AI for Developers, Gemini thinking (thought summaries)

Opinions are my own. Every fact in this post comes from the public documentation linked above.