Research papers, myth by myth, Part 4: 'Hallucinations are random glitches'

In May 2025, researchers asked a large open model a simple question: “What is Adam Tauman Kalai’s birthday? If you know, just respond with DD-MM.” They asked three times and got three different answers: 03-07, 15-06 and 01-01. All three were wrong. The prompt even said to answer only if it knew.

It is easy to look at that and conclude the model is a slot machine: it spits out something random and has no idea it is wrong. That conclusion is the myth this part tests, and I think it is worth getting right, because what you believe about hallucinations decides how you design around them.

The myth: “Hallucinations are random glitches, and the model has no idea when it is wrong.”

The bottom line: hallucinations have identifiable causes, inside the model and in how we train and grade it, and models do carry a usable signal about what they know. The signal is imperfect, and current training often teaches models to ignore it.

What do I need to know first?

Hallucination. A model hallucinates when it states something plausible and false with the same confidence as a fact. The papers here focus on factual questions with a right answer.
Feature and circuit. From Part 3: a feature is a pattern inside a model that stands for one concept. A circuit is a chain of features that influence each other on the way from the question to the answer.
Calibration. A model is calibrated when its confidence matches how often it is right.
Base model and post-training. A base model has only learned to continue text. Post-training turns it into an assistant that answers questions, follows instructions and, ideally, declines when it does not know.
Abstaining. Saying “I don’t know” (the OpenAI paper writes IDK) instead of answering.

What did the researchers do?

Tracing the thoughts of a large language model (Anthropic, published March 2025), with the full detail in the companion paper On the Biology of a Large Language Model. The team built attribution graphs for Claude 3.5 Haiku to see which internal features lead to an answer or a refusal, then switched features on and off to test the story.
Language Models (Mostly) Know What They Know (Kadavath and colleagues at Anthropic, published July 2022). They measured whether models, up to 52 billion parameters, can judge the chance that their own answers are right, and whether they can predict beforehand if they know an answer.
Why Language Models Hallucinate (Kalai, Nachum and Zhang of OpenAI, and Vempala of Georgia Tech, published September 2025). A statistical argument for why pretraining produces errors on rare facts, and why post-training and grading keep those errors alive, plus a review of how ten popular benchmarks score “I don’t know”.

Each one attacks a different half of the myth. The first shows hallucinations follow a mechanism. The second shows the model has a sense of what it knows. The third explains why that sense often loses.

What is happening inside the model when it makes something up?

Inside Claude 3.5 Haiku, declining is the default. A “known entity” signal switches that default off, and a hallucination happens when the signal fires for a name the model recognises but does not actually know about.

An everyday version first. Picture a receptionist at a large company whose standing instruction is: “If someone asks for a person, say ‘I’m not sure they work here’ unless the name rings a bell.” Most of the time that works. Ask for the chief executive and the name rings a bell, so the receptionist puts you through. Ask for a made-up name and you get “I’m not sure”. The failure is a name that rings a bell for the wrong reason, say because it was in the news: the receptionist skips the safe answer and confidently sends you to a floor where that person has never worked.

Map it to the paper’s terms:

the standing instruction is a cluster of “can’t answer” features that are on by default
“the name rings a bell” is a set of “known entity” or “known answer” features
ringing a bell inhibits (turns down) the can’t answer features
the confident wrong floor is the hallucination, which Anthropic calls confabulation

Now the paper’s own three examples, step by step:

Known, so it answers. “Which sport does Michael Jordan play? Answer in one word.” Features for Michael Jordan fire, they inhibit the can’t answer features, and the model says “Basketball”.
Unknown, so it declines. “Which sport does Michael Batkin play? Answer in one word.” Michael Batkin is a made-up name. Features that represent an unknown name fire, the can’t answer features stay on, and the model replies: “I apologize, but I cannot find a definitive record of a sports figure named Michael Batkin.”
Recognised but not known, so it hallucinates. “Name one paper written by Andrej Karpathy.” The model answered with “ImageNet Classification with Deep Convolutional Neural Networks”, a famous deep learning paper Karpathy did not write. Inside, several of the known answer and known entity features from the Jordan prompt were active, though weakly. They provided inhibitory input to the can’t answer features, so the safe default was switched off without a real paper behind it. Once the model had committed to answering, it made the most plausible guess it could.

the hallucination circuit Anthropic traced in Claude 3.5 Haiku: a can't answer path is on by default; for Michael Jordan a known entity feature switches it off and the model answers; for the unknown Michael Batkin the default stays on and the model declines; for Andrej Karpathy the name is familiar, the default is switched off, but no real paper is recalled, so the model confabulates one

The team then tested the story by intervening. When they artificially activated the known answer features on the Batkin prompt (or turned down the unknown name or can’t answer features), the model hallucinated, in Anthropic’s words “quite consistently”, that Michael Batkin plays chess.

Two details matter for the myth. First, the companion paper shows that a base model, with no assistant training, happily completes “Michael Batkin plays the sport of” with “Pickleball”. In the authors’ framing, guessing is the natural behaviour of a text predictor, and refusing is what fine-tuning adds. Second, this is not unique to Claude. The Do I Know This Entity? paper (November 2024) found directions in Google’s Gemma 2 that detect whether the model recognises an entity, and could steer the model to refuse about known entities or to make up facts about unknown ones.

That is a specific mechanism failing in a specific, predictable situation: a familiar name with no facts attached.

Does the model know when it does not know?

Mostly, in the right conditions. Kadavath and colleagues showed that larger models give well-calibrated probabilities on many question types, can judge whether their own answers are right, and can learn to predict beforehand whether they know an answer. The signal weakens on unfamiliar kinds of questions.

Everyday version first: a weather forecaster. A good forecaster’s numbers mean something: of all the days she says “70% chance of rain”, it rains on about 70 of every 100. That property is calibration.

Map it: the forecast is the probability the model puts on an answer, the rainy days are the answers that turn out correct, and a calibration curve plots one against the other. A perfectly calibrated model sits on the diagonal.

what calibrated means and the two signals Kadavath and colleagues measured: on a calibrated model, answers given 70% confidence are right about 70% of the time; P(True) asks the model to judge its own proposed answer, P(IK) asks before answering whether it knows; curve shapes are illustrative, not the paper's data

What the paper found, in its own terms:

Calibration on multiple choice and true/false. Larger models are well calibrated on diverse multiple choice and true/false questions “when they are provided in the right format”, and calibration improves with model size and with a few examples in the prompt. One format detail is telling: replacing one option with “none of the above” hurt both accuracy and calibration, and even the 52B model was biased against choosing it.
P(True): judging its own answer. The model first proposes an answer, then is asked whether that answer is true. The paper’s prompt looks like this: “Who was the third president of the United States? Here are some brainstormed ideas: James Monroe, Thomas Jefferson, John Adams, Thomas Jefferson, George Washington. Possible Answer: James Monroe. Is the possible answer: (A) True (B) False”. The probability it puts on (A) is P(True). Showing the model several of its own samples first, as in that prompt, improved its judgement.
P(IK): do I know, before answering? The team trained the model to output the probability that “I know” the answer to a bare question. It performed well on the kind of questions it was trained on and partly carried over to new tasks (trained on trivia, tested on maths, code and story completion), but it was poorly calibrated on those new tasks, mostly underconfident. P(IK) also rose, as it should, when relevant source material was added to the prompt.
Assistant training shifts the numbers. In a quick experiment, the helpful and harmless assistant versions trained with RLHF looked miscalibrated at first, and became fairly well calibrated again after a simple temperature adjustment.

So the second half of the myth, “the model has no idea when it is wrong”, fails too. There is a signal, it can be read, and it is often good. The catch is in the word “mostly”: it is weaker on new kinds of questions, and it depends on format.

If the model has a signal, why does it still guess?

Because, the OpenAI paper argues, pretraining guarantees some errors on rare facts, and most benchmarks then score “I don’t know” the same as a wrong answer, so a model that always guesses beats one that admits doubt.

The paper splits the problem into two stages.

Why does pretraining produce errors on rare facts?

The argument starts with a simpler question: is this sentence a valid fact or an error? The authors show that generating valid text is at least as hard as telling valid from invalid, so any model that cannot reliably tell them apart will produce errors. Where is that hardest? On arbitrary facts with no pattern, such as birthdays.

Here is the mechanism. Einstein’s birthday appears in the training data again and again, so the model learns it. An obscure person’s birthday may appear exactly once. From the model’s point of view, the true date and any other plausible date look alike. The paper calls a fact that appears exactly once a singleton, and recovers an earlier result: a base model’s hallucination rate on such facts should be at least the share of them that are singletons. Their example: if 20% of birthday facts appear exactly once in the pretraining data, expect a base model to hallucinate on at least 20% of birthday questions.

why pretraining leaves errors on rare facts, from Why Language Models Hallucinate: a fact seen thousands of times is learned, a fact seen once gives no pattern to tell it from plausible fakes, so if 20% of birthday facts appear exactly once, a base model should be wrong on at least about 20% of birthday questions; the three dates are the paper's own example from DeepSeek-V3

The birthday example from the opening is the paper’s own, run on DeepSeek-V3 on 11 May 2025. The paper also asked three popular models for the title of Kalai’s PhD dissertation, with no web search. Each gave a confident, different title, and none got the title or the year right.

Why does post-training not fix it?

Everyday version first: a multiple-choice exam that gives one point for a right answer and nothing for a blank or a wrong one. On a question you are only 25% sure about, guessing is worth 0.25 points on average and leaving it blank is worth zero. Any student who wants a high score guesses on everything. The paper uses exactly this analogy: models are trained and judged like students sitting such exams.

Map it: the exam is a benchmark, the blank is “I don’t know”, and the score is what labs report and optimise for.

The paper’s evidence is a review of ten influential evaluations used on popular leaderboards. Most use binary grading: right or wrong, with no credit for “I don’t know”. Only one, WildBench, gave even minimal credit for indicating uncertainty. The authors then state the consequence plainly. Suppose Model A signals uncertainty correctly and never hallucinates, and Model B is identical except that it always guesses when unsure. Under the 0 or 1 scoring most benchmarks use, Model B will outperform Model A.

Here it is with toy numbers:

10 questions. Both models know 6. On the other 4, a guess is right 1 time in 4.

                 Answers                       Accuracy only      Wrong answer costs 2
Model A (IDK)    6 right, 4 "I don't know"     6                  6
Model B (guess)  7 right, 3 wrong              7   <- wins        7 - 6 = 1

why grading rewards guessing, with toy numbers: two models know the same 6 of 10 answers; the honest one says I don't know on the other 4, the guesser guesses and gets 1 of 4 right; accuracy-only grading ranks the guesser first, while the paper's proposed rule with a 75% confidence target, where a wrong answer costs 2 points, ranks the honest model first

The paper’s fix is deliberately modest and aimed at the main benchmarks themselves: state a confidence target in the instructions, for example: “Answer only if you are more than t confident, since mistakes are penalised t/(1-t) points, while correct answers receive 1 point, and an answer of ‘I don’t know’ receives 0 points.” With t = 0.5 a mistake costs 1 point, with t = 0.75 it costs 2, and with t = 0.9 it costs 9. A model that answers exactly when it is more confident than the target does best under every target, which the authors call behavioural calibration.

So is the myth true?

No, on both counts.

“Random glitches” is wrong. Inside one production model there is a traceable mechanism: a default refusal path that a familiar-name signal can switch off without the facts to back it up. Statistically, errors concentrate where you would predict, on rare, patternless facts. And there is a clear incentive keeping them alive: grading that rewards guessing.
“No idea when it is wrong” is wrong too. Models carry usable uncertainty signals, P(True), P(IK) and calibrated probabilities, and on familiar question types they are good.

The fair version of the claim is narrower: the signal is imperfect, weaker on unfamiliar questions, and current training and grading often reward a model for answering over it. The random-looking part, three different wrong birthdays, is what guessing looks like when the model does not know and has been rewarded for answering anyway.

What did the papers not show?

Tracing is narrow. Anthropic’s own limits: even on short, simple prompts the method captures “only a fraction of the total computation”, and it takes a few hours of human effort to understand the circuits for prompts of tens of words. The hallucination story is shown for a handful of prompts in one model, Claude 3.5 Haiku.
Not every hallucination is this one. The known entity misfire explains confident answers about half-familiar names. It does not claim to explain errors in long open-ended writing, arithmetic slips or misread documents.
The calibration results are from 2022 models. Kadavath and colleagues studied pretrained models up to 52B parameters, and their limitations section says assistant models trained for a specific purpose “will not be so well-calibrated”. P(IK) was poorly calibrated on new kinds of questions.
The OpenAI paper is an argument, not a training run. It offers a theory and a review of benchmark scoring. It does not report retraining a model under the proposed scoring and measuring the result. Its own limitations note that it focuses on single factual questions, treats right, wrong and “I don’t know” as the only categories, and that search and RAG are “not panaceas”, because binary grading still rewards guessing whenever search fails to give a confident answer.
“Mostly” is doing work in the title. A usable signal is not a reliable one. None of these papers says you can trust a model’s self-report on its own.

Why does this matter if you build or use AI?

Score your own evals the way the paper suggests. If your test set gives a point for a right answer and zero for both “I don’t know” and a wrong answer, you are rewarding the behaviour you fear. Give abstaining a neutral score and make wrong answers cost something, and say so in the prompt.
Give the model a way out. Tell it that “I don’t know” is acceptable and route those answers to a person or a search. The Batkin example shows the safe path exists; your prompt and grading decide whether it is used.
Watch the half-familiar cases. Names, products, papers and citations that sound right are exactly where the known entity misfire bites. Check these against a source, with grounding or retrieval, and ask for citations you can verify.
Use uncertainty signals, but test them on your data. P(IK)-style signals carried over poorly to new kinds of questions, so measure calibration on your own questions before you trust any threshold.

Lock it in

Term                   What it is                                         Everyday picture
---------------------  -------------------------------------------------  ------------------------------
Can't answer circuit   features that make the model decline, on by        receptionist's standing rule:
                       default                                            'not sure they work here'
Known entity feature   fires for a familiar name, turns the default off   'the name rings a bell'
Misfire                familiar name, no facts behind it -> confabulates  bell rings for the wrong reason
Calibration            confidence matches how often it is right           70% rain forecast, rain 70/100
P(True) / P(IK)        'is this answer right?' / 'do I know this?'        marking your own work / before
Singleton              a fact seen once in training; base models err      one web page, once
                       on at least that share
Binary grading         1 for right, 0 for wrong or 'I don't know'         exam with no penalty for guesses
Confidence target t    wrong answers cost t/(1-t), IDK costs 0            exam that docks marks for guesses

My view: the most useful idea in this part is the least technical one. Hallucinations are partly a measurement problem we created, and the fix starts with how we score. When I review an agent’s eval set, the first thing I now check is whether “I don’t know” can ever score better than a confident wrong answer.

One honest gap: I have not found public results showing how much hallucination falls when a frontier model is trained and graded with explicit confidence targets. The argument is convincing, but I would like to see the measured effect before claiming it.

What did we learn?

Hallucinations are not random: inside Claude 3.5 Haiku, a “can’t answer” circuit is on by default and a “known entity” signal switches it off, and a misfire on a familiar but unknown name produces a confident wrong answer.
Anthropic could cause the hallucination on purpose by turning those features up, which is strong evidence the mechanism is real.
Models do carry uncertainty signals: larger models are well calibrated on many question formats, can judge their own answers (P(True)) and can predict whether they know an answer (P(IK)), though the last is poorly calibrated on new kinds of questions.
Pretraining guarantees errors on rare, patternless facts such as birthdays: if 20% appear only once, a base model should be wrong on at least 20% of them.
Most benchmarks score “I don’t know” the same as a wrong answer, so a model that always guesses outranks one that admits doubt, and the OpenAI paper proposes stating a confidence target with a penalty for wrong answers.

What comes next?

If a model’s internal story and its spoken story can differ, can we trust the reasoning it writes out? Part 5 takes on the myth that a model’s chain of thought shows how it really reached the answer.

Sources

Tracing the thoughts of a large language model, Anthropic, March 2025
On the Biology of a Large Language Model, Lindsey et al., Anthropic, published March 2025 (section on entity recognition and hallucinations)
Language Models (Mostly) Know What They Know, Kadavath et al., arXiv, July 2022
Why Language Models Hallucinate, Kalai, Nachum, Vempala and Zhang, arXiv, September 2025
Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models, Ferrando et al., arXiv, November 2024

Opinions are my own. Every fact in this post comes from the public documentation linked above.