Research papers, myth by myth, Part 7: 'More context is always better'

Every few months a model launches with a bigger context window. In 2023 a typical window held about 4,000 tokens, roughly a long essay. Within a year some models accepted a million or more, several long novels at once. The natural conclusion is that we can stop being picky: put every document, every example and the whole chat history in the prompt, and let the model sort it out.

The myth this post tests is:

“A bigger context window means the model reads everything equally well, and more examples are always safer.”

Bottom line: a long context is a capability and an attack surface at the same time. Where you put information changes how well the model uses it, and the same ability that lets a model learn from hundreds of examples lets an attacker teach it to misbehave. Two papers make the case, and a third shows the helpful side of the same mechanism.

What do I need to know first?

Picture a student revising from a 200-page pack the night before an exam. They read the first few pages carefully, skim the middle, and read the last pages again just before bed. Ask them about page 3 or page 198 and they answer well. Ask about page 100 and they are shakier, even though they did, technically, read it.

Mapped to the real terms:

The revision pack is the context window: everything the model is given in one call.
Reading the start carefully is primacy bias: better use of information near the beginning.
Re-reading the end is recency bias: better use of information near the end, closest to the question.
The skimmed middle is the problem Liu and colleagues named “lost in the middle”.

Two more terms come up:

In-context learning means a model picks up a task from examples placed in the prompt, without any change to its weights. Show it three English and French pairs and it translates the fourth. A shot is one such example; few-shot means a handful, many-shot means hundreds or thousands.
A jailbreak is a prompt that gets a model to do something its safety training should stop it doing.

What did the researchers do?

Paper 1: Lost in the Middle. Nelson Liu and colleagues at Stanford, UC Berkeley and Samaya AI published Lost in the Middle: How Language Models Use Long Contexts in July 2023 (revised November 2023). They tested models that were current then: GPT-3.5-Turbo, Claude-1.3, MPT-30B-Instruct and LongChat-13B, with a smaller GPT-4 check in an appendix.

The main test was multi-document question answering. Each prompt held a real search question from the NaturalQuestions dataset plus 10, 20 or 30 Wikipedia passages. Exactly one passage contained the answer; the rest were distractors: passages on the same topic, picked by a retrieval system, that do not contain the answer. The researchers kept everything fixed and moved only the answer passage: first, fifth, tenth, and so on to last. A second, synthetic test gave models a list of JSON key and value pairs and asked for the value of one key.

Paper 2: Many-shot jailbreaking. Cem Anil and colleagues at Anthropic, with university co-authors, published Many-shot Jailbreaking in April 2024. Anthropic briefed other AI developers before publishing. They tested Claude 2.0, GPT-3.5, GPT-4, Llama 2 (70B) and Mistral 7B by filling one prompt with fake dialogues between a user and an assistant, up to 256 of them, then adding a final harmful question.

Paper 3, the helpful side: Many-Shot In-Context Learning. Rishabh Agarwal and colleagues at Google DeepMind published Many-Shot In-Context Learning in April 2024. With Gemini 1.5 Pro and its one-million-token window they scaled the number of examples in the prompt from a few to hundreds or thousands (up to 8,192 shots) across translation, summarisation, maths, planning and sentiment tasks.

What did they find?

Finding 1: the middle of the prompt is a weak spot

Accuracy followed a U shape: high when the answer was at the start, high again at the end, lowest in the middle. Here are GPT-3.5-Turbo’s numbers with 20 documents, from the paper’s appendix table:

Answer passage at position 1    75.8%
Answer passage at position 5    57.2%
Answer passage at position 10   53.8%   <- the middle
Answer passage at position 15   55.4%
Answer passage at position 20   63.2%
No passages at all (closed book) 56.1%

Step by step, that says three things:

Moving the answer from first to tenth place, with nothing else changed, dropped accuracy by 22 points (75.8 minus 53.8).
At position 10, the model did worse than when it got no documents at all and answered from memory (53.8% against 56.1%). Giving it the right passage in the wrong place was worse than giving it nothing.
The end recovered some ground (63.2%) but not all of it: primacy was stronger than recency for this model.

Lost in the Middle, multi-document question answering with 20 documents: GPT-3.5-Turbo scores 75.8% when the answer document comes first, 53.8% when it sits in the middle and 63.2% when it comes last, below its 56.1% with no documents at all

Some more details matter. Claude-1.3 had a flatter curve (59.9% at the start, 55.9% at position 5, 60.1% at the end) but the same dip. The “extended context” versions of models, GPT-3.5-Turbo (16K) and Claude-1.3 (100K), scored almost exactly the same as the standard versions: a bigger window did not mean better use of it. GPT-4 scored higher overall but still showed the U shape on a 500-question subset.

A practical result sat in a later section. When the researchers retrieved passages for open questions and gave the model more of them, going from 20 to 50 passages improved GPT-3.5-Turbo by only about 1.5 points and Claude-1.3 by about 1 point, while making each prompt much longer, slower and more expensive. The authors suggest reranking (putting the best passages first) and retrieving fewer passages when that is enough.

Finding 2: hundreds of fake examples can override safety training

The attack is simple: write many fake dialogues in which an assistant happily answers harmful questions, put them all in one prompt, and finish with the question you actually want answered. Anthropic’s own illustration uses lock picking as the fake shot and a bomb question as the target. I will keep to placeholders:

User: <harmful question 1>
Assistant: Sure, here is how... <detailed answer>
User: <harmful question 2>
Assistant: Sure, here is how... <detailed answer>
... (hundreds more)
User: <the question the model is trained to refuse>
Assistant:

With one fake dialogue, or a handful, the safety training holds and the model refuses. On Claude 2.0, across tasks from insulting users to violent and deceptive content, the paper reports that the attack “doesn’t work at all with 5 shots” but “works consistently with 256 shots”.

Many-shot jailbreaking: the attacker fills one prompt with hundreds of fake dialogues in which an assistant complies, then asks the real question; on Claude 2.0 the attack does not work at all with 5 shots and works consistently with 256

Three findings make this more than a trick:

It follows a power law. A power law here means that each doubling of the number of shots improves the attack by a predictable amount, so the trend can be forecast. The authors found the trend held up to hundreds of shots.
Larger models are often more vulnerable. Bigger models tend to be better in-context learners, so on many tasks they pick up the harmful pattern from fewer shots.
Simple fixes only delayed it. Anthropic reports that fine-tuning the model to refuse such prompts meant it took more shots before harmful answers appeared, but they eventually appeared. Changing the prompt format shifted the curve without changing its slope.

What worked better was changing the prompt before the model sees it. In the paper’s test, adding demonstrations of refusals at the start cut attack success only from 61% to 54% on a 205-shot prompt in the deception category, while adding warning text both before and after the prompt cut it to 2%. Anthropic’s blog describes its deployed approach as classifying and modifying the prompt before it reaches the model, and notes the trade-off with usefulness still needs study.

Finding 3: it is the same mechanism that makes many examples useful

The jailbreak works because in-context learning works. Anthropic showed that ordinary, harmless in-context learning tasks improve with more shots following the same kind of power law as the attack.

Google DeepMind’s paper shows the bright side in detail. Translating English into Bemba, a language with little training data, Gemini 1.5 Pro improved steadily as the prompt grew to 997 example sentence pairs (about 85,000 tokens), and beat the previous best published score of 35 (chrF2++, a translation quality score) set by NLLB, a dedicated translation system. In a sentiment task where the labels were deliberately flipped (positive sentences labelled negative and the other way round), a few examples could not overcome what the model learnt in pre-training, but with enough examples it learnt the flipped rule.

Same mechanism, two uses: in-context learning lets hundreds of examples teach a model a new task such as English to Bemba translation, and the same mechanism lets hundreds of fake dialogues teach it to drop its refusals

Put the two side by side and the lesson is plain: the capability that lets you teach a model a niche task in the prompt is the same capability an attacker uses. You cannot keep one and remove the other just by picking a bigger model.

So is the myth true?

False, on both halves.

“The model reads everything equally well” is false for the models tested. Position mattered by more than 20 points, and a bigger window did not fix it.
“More examples are always safer” is false. More examples are often better for accuracy, but Agarwal and colleagues also saw performance fall again past a point on some tasks (on MATH and GPQA, accuracy dropped after 125 shots), and in the wrong hands more examples are exactly what makes the jailbreak work.

The fair version is: a long context is a powerful tool that needs the same care as any other input channel. Order your content, measure what the model actually uses, and treat long, untrusted prompts as something to screen.

What did the papers not show?

Lost in the Middle tested 2023 models. Newer models have improved a lot on long-context retrieval. Google’s Gemini 1.5 report (March 2024) reports near-perfect recall (above 99%) on long-context retrieval tests up to at least 10 million tokens. Those “needle in a haystack” tests are easier than reasoning over many similar passages, so the U shape is a reason to test your own model, not proof that today’s models still show it at the same size.
It used one controlled set-up. Exactly one passage held the answer and the rest were plausible distractors. Real documents can contain the answer in several places, or nowhere.
The many-shot jailbreak paper measured mitigations in a lab. The 61% to 2% result was on one category and one prompt length; the authors call for more work on the safety and usefulness trade-off.
Neither paper fully explains why. Liu and colleagues ran preliminary checks on architecture and instruction tuning; Agarwal and colleagues state that they do not completely understand why performance sometimes drops with more examples.

Why does this matter if you build or use AI?

Put the most important material at the start or the end. Instructions and the question near the end, the best evidence first. If you use RAG, rerank and keep the top few passages instead of pasting fifty.
Do not assume more retrieved text means better answers. The paper’s own result was 1 to 1.5 points for 2.5 times the passages, at a real cost in latency and price.
Test position on your own model. Move a known fact through a long prompt and measure. It takes an afternoon and replaces assumptions with data.
Treat long, untrusted input as an attack surface. Uploaded files, pasted chat logs and tool results can carry hundreds of fake dialogues. Screen inputs with guardrails before they reach the model, and include many-shot attacks in your red-team tests.
Use many-shot prompting on purpose. For a niche format or a low-resource language, a few hundred good examples can be a cheaper first step than fine-tuning. My Production-ready agents series covers how to build the evaluations that tell you whether it worked.

Lock it in

Context window      How much the model can read in one call, not how well it uses it
Lost in the middle  Answer at the start or end: best; in the middle: worst
                    GPT-3.5-Turbo, 20 docs: 75.8% / 53.8% / 63.2% (2023 models)
In-context learning Learning a task from examples in the prompt, weights unchanged
Many-shot learning  Hundreds of examples: big gains on niche tasks (Gemini 1.5 Pro)
Many-shot jailbreak Hundreds of fake compliant dialogues override refusals
                    Claude 2.0: fails at 5 shots, works at 256; follows a power law
Same mechanism      The ability to learn from examples is the vulnerability
Defence             Order content well; screen long untrusted prompts before the model

My view: I think “how big is the context window?” is the wrong first question when choosing a model for long documents. The better question is “how well does it use a fact at token 400,000 when there are 50 similar facts around it?”, and the only honest answer comes from your own test.

One honest gap: I could not find a public, like-for-like rerun of the exact Lost in the Middle test on today’s frontier models. The improvement in long-context retrieval is well documented, but whether the middle dip has gone, shrunk or simply moved to longer prompts is something I would measure rather than assume.

What did we learn?

A bigger context window lets a model read more, but the models tested in 2023 used information at the start and end of the prompt far better than information in the middle.
In Lost in the Middle, moving one answer passage from first to tenth place cut GPT-3.5-Turbo’s accuracy from 75.8% to 53.8%, lower than with no documents at all.
Many-shot jailbreaking fills a prompt with hundreds of fake compliant dialogues; on Claude 2.0 it failed at 5 shots and worked consistently at 256, following a power law.
The jailbreak and helpful many-shot learning are the same mechanism, in-context learning, which is why larger, better learners can be more vulnerable.
Practical defences are ordering and trimming what goes into the prompt, testing position on your own model, and screening long untrusted inputs before they reach the model.

What comes next?

Part 8 takes on another comfortable belief: that training a model on human feedback teaches it to be right. The InstructGPT and Anthropic papers show what feedback does well, and the sycophancy papers show what it quietly rewards instead. Read Part 8: ‘Human feedback teaches a model the truth’.

Sources

Lost in the Middle: How Language Models Use Long Contexts, Liu et al., arXiv 2307.03172, published July 2023, revised November 2023
Many-shot jailbreaking, Anthropic research post, published April 2024
Many-shot Jailbreaking (paper PDF), Anil et al., Anthropic, published April 2024
Many-Shot In-Context Learning, Agarwal et al., Google DeepMind, arXiv 2404.11018, published April 2024
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, Gemini Team, Google, arXiv 2403.05530, published March 2024

Opinions are my own. Every fact in this post comes from the public documentation linked above.