Sheet 02 / 05 — Blogs← Archive

Production-ready agents, Prerequisite 3: How a model works: tokens, context, settings and what we never ask it to do

The model basics every later part assumes: how training sets the weights, and what neurons, features and superposition are; what a token is and why it sets price, limits and speed; how a model writes one token at a time, and why chain-of-thought and thinking help; what fills the context window and why the model remembers nothing between calls; what temperature, top-k, top-p and the other settings really do (and why Google now says to leave them alone); what comes back besides text; and the jobs this series never gives to a model.

28 October 2026· 38 min read · ~55 min listen· Agents · Models · Gemini · Fundamentals

Part 1 defines a model in three short paragraphs and moves on. That is enough to start, but almost every later part leans on ideas it never explains: tokens, context windows, temperature, thinking, structured output, finish reasons. Readers new to this have told me those words are where they get lost.

So this prerequisite fills the gap. It explains how a model works from the ground up, using Gemini’s public documentation, and ends with the list of jobs this series never gives to a model, and why. If you already know what top-p does and why a model has no memory, skip to the last section; it is the one the rest of the series depends on.

If you prefer to watch first, here is the heart of this prerequisite as a short animated explainer.

Video: how a model works, from tokens to the jobs we never give it, in three minutes. The narration is a synthetic voice; the script and animation are mine.

Where are we in the architecture?

This prerequisite does not add a box. It explains the Gemini models box on the right: tokens, context, settings, and the jobs the rest of the picture takes away from the model. Part 1.25 walks through the whole picture, layer by layer.

the complete Acme architecture: Maria asks in Gemini Enterprise; the router runs on Agent Runtime behind plugins and Model Armor and calls the marketing and supply agents over A2A; they reach BigQuery and Databricks through Agent Gateway; identity and secrets on the left, models, knowledge and the market watch managed agent for public research on the right, traces, logs and analytics below; above it, shoppers reach Gemini Enterprise for CX, whose conversation insights land in BigQuery, and the Legal industry edition plugs into Gemini Enterprise

What is a model, really?

The bottom line: a model is a very large next-word predictor whose knowledge was fixed when training ended; using it does not teach it anything.

Start with the predictive text on your phone. Type “see you” and it offers “soon”, “tomorrow”, “later”, because it has seen those words follow “see you” many times. Now imagine that keyboard trained on an enormous amount of text, big enough to continue whole paragraphs, answer questions and write code. It is still predicting what comes next. And like a student who left school in a given year, it knows what it was taught up to then; chatting with it afterwards does not add to its schooling.

A large language model (LLM) is a very large mathematical function. It takes text (and, for most current models, images, audio and video) and produces a prediction of what comes next. Its behaviour is stored in parameters, also called weights: billions of numbers set during training, when the model was shown enormous amounts of text and adjusted, step by step, to predict it better.

Using a trained model is called inference. Every call to Gemini in this series is inference: the weights do not change, and nothing you send teaches the model anything. (Changing the weights is tuning, which Part 14 covers.) The keyboard is the model, its years of reading are training, and each suggestion is inference. How does this apply to Acme? Two consequences follow:

A model only knows what was in its training data, up to a knowledge cutoff date. Anything newer, and anything private to your company, must reach it through the prompt or a tool. Ask a model with no tools “What was Acme Pure’s OTIF in the week of 1 June 2026?” and it cannot know: that figure, 85.91%, exists only in supply’s supply_service table, so it has to come from a tool that reads it.
Models come in families and sizes. A large model is usually more capable and slower; a small one is faster and cheaper. Part 13 covers how to choose.

What is inside a model? Weights, neurons and features

A model’s knowledge lives in its weights, the numbers that training sets. Neurons are where those numbers do their work, and features, the concepts a model actually uses, are patterns spread across many neurons. If you keep one line from this section, keep the last one: one neuron is not one idea.

How does training set the weights?

Training is one small loop, repeated an enormous number of times:

Show the model a stretch of real text with the next token hidden: “The Golden Gate Bridge is in San …”
Predict. The model gives every token in its vocabulary a probability. Say it gives “ Francisco“ 0.70.
Measure the loss. The loss is one number for how wrong the prediction was. For next-token prediction the usual loss is minus the logarithm of the probability given to the right token: -ln 0.70 ≈ 0.36. A certain, correct prediction would score 0.
Nudge the weights. An algorithm called gradient descent works out which way each weight should move to make the loss a little smaller, and moves every weight a tiny step that way. Next time, “ Francisco“ might get 0.75, a loss of about 0.29.
Repeat with the next stretch of text, billions of times. Some text is held back and never trained on, so the builders can check that the model learned patterns rather than memorising pages.

To see one nudge with real numbers, shrink the model to a single weight. A weight (w) says how strongly an input counts; a bias (b) is a baseline added on top. This toy uses squared error as its loss rather than the log loss above, but the nudge works the same way:

y = w.x + b        w = 0.72   b = 0.10   input x = 2.0   right answer = 1.70

predict   y = 0.72 × 2.0 + 0.10  = 1.54        off by 0.16
loss      (1.54 - 1.70)²         = 0.0256
nudge     w: 0.72 -> 0.752   b: 0.10 -> 0.116   (step size 0.05, in the direction that lowers the loss)
predict   y = 0.752 × 2.0 + 0.116 = 1.62       off by 0.08
loss      (1.62 - 1.70)²          = 0.0064      a quarter of what it was

A language model does exactly this, with billions of weights moved together on every step. That is all parameters means: the weights and biases, counted. Google’s open Gemma 4 models, for example, come in sizes up to 31 billion parameters, stored by default at 16-bit precision, two bytes each, so the largest is roughly 62 GB of numbers. Google does not publish parameter counts for Gemini. When training ends the numbers are frozen: inference only reads them, which is why nothing you send teaches the model anything.

What are neurons and features?

Where people get tripped up is the difference between weights, neurons and features. Think of a piano:

The strings are the weights and biases. Their tension is set at the factory (training) and does not change while you play (inference).
The keys are the neurons. A neuron is one small calculation inside a layer: multiply its inputs by its weights, add its bias, pass the result through a simple function and hand on one number, its activation. Press a key and it sounds.
The chords are the features. A feature is a pattern of activity across many neurons that stands for one concept, such as the Golden Gate Bridge or code with a security hole. A chord means something no single key does.

what is inside a model, as a piano: the weights and biases are the strings, tuned during training and then frozen; neurons are the keys, and one key plays in many chords; features are the chords, patterns across many neurons that stand for one concept; because a model needs more features than it has neurons, features share neurons, which is called superposition; key names and chords are illustrative

The surprise is that one key plays in many chords. Anthropic’s Toy Models of Superposition (September 2022) puts the idea in one line: networks “want to represent more features than they have neurons”, so they pack several features into the same neurons. This is called superposition. The result is that many neurons are polysemantic, “responding to several unrelated features”: look at one neuron on its own and it seems to fire for a jumble of unrelated things. In the paper’s small trained networks, both clean single-concept neurons and polysemantic ones form.

So researchers stopped reading neurons one at a time and went looking for the chords. In Scaling Monosemanticity (May 2024), Anthropic trained a tool called a sparse autoencoder on the middle layer of Claude 3 Sonnet and pulled out dictionaries of up to about 34 million features. It treats a feature as one of the “directions in their activation spaces”: a weighted pattern across many internal numbers at once, the chord rather than the key. Among the features it reports are ones for the Golden Gate Bridge, for security vulnerabilities in code and for sycophancy.

Why would a model that only predicts the next token build a Golden Gate Bridge chord? My view: because it is the cheapest way to predict well. To continue “I drove north out of San Francisco across the famous red suspension bridge into Marin County, looking out over the …”, it helps far more to recognise which bridge than to memorise the sentence. That is the best intuition I know for why plain next-token prediction produces something that looks like understanding.

For Acme, the practical point is what is not in the strings. Acme Pure’s 85.91% OTIF lives in supply’s supply_service table; no weight or feature in Gemini holds it, which is why the supply agent fetches it with a tool. Features are a research tool for model builders, not a setting you get in the Gemini API. Prerequisite 4 picks this up in its interpretability section: how turning one feature up changes a model’s behaviour, and why none of it is what you show an auditor.

To lock it in:

TERM              WHAT IT IS                                          ON THE PIANO
weights, biases   numbers set in training, frozen at inference        the strings and their tension
parameters        all the weights and biases, counted                 every string in the piano
neuron            one calculation that outputs one number             one key
feature           a pattern across many neurons = one concept         a chord
superposition     more features than neurons, so they share neurons   one key in many chords

What is a token?

The bottom line: a model reads and writes chunks called tokens, often smaller than a word, and almost everything you pay for or run out of is counted in them.

Think of a box of building bricks. Common shapes come as one big piece; an unusual shape is built from several small ones. Tokenization works the same way: a common word is often one token, while a rare word or a product code is built from several (the exact split depends on the model’s vocabulary). And just as a builder is charged per brick, not per finished model, you are charged per token, not per word.

A model does not read letters or words. It reads tokens. Google’s token guide explains: “Tokens can be single characters like z or whole words like cat. Long words are broken up into several tokens.” The full set of tokens a model knows is its vocabulary, and splitting text into tokens is tokenization.

one sentence split into tokens, for illustration: words, parts of words, spaces and punctuation each become a token; as a rule of thumb a token is about four characters of English, and 100 tokens are about 60 to 80 words; images, audio and video are turned into tokens too

Three rules of thumb from the same guide are worth remembering:

For Gemini models, “a token is equivalent to about 4 characters”, and “100 tokens is equal to about 60-80 English words”.
Everything is tokenized, not just text. The guide gives 258 tokens for an image up to 384 pixels on each side (larger images are cut into tiles of 258 tokens each), 263 tokens per second of video and 32 tokens per second of audio.
You can measure instead of guessing: count_tokens returns the size of a request before you send it, and every response reports its usage: input, output, thinking, cached and tool-use tokens.

Tokens matter because almost everything is measured in them. Prices are per million input and output tokens (Part 11 and Part 13). Limits are in tokens: the context window and the maximum output. And latency grows with the number of tokens the model has to read and, above all, write.

How does this apply to Acme? By the rule of thumb, Maria’s question “What did the Acme Pure Refill Launch cost us in Western Europe?”, 63 characters, is roughly 16 tokens, while one 151-row tool result from campaign_performance is thousands (the context window section below estimates it). The bricks that cost Acme money are the tool results, not the questions.

How does a model write an answer?

One token at a time. The model reads everything in its context, gives every token in its vocabulary a probability of coming next, picks one, adds it to the context, and does it again.

Picture writing a story in pen with a rule: before each word, you reread everything on the page, list the words that could come next with how likely each feels, choose one, and write it down. You cannot rub anything out, and you stop when you write “The End” or run out of paper. The page is the context, the list of likely words is the probability for each token, “The End” is the stop token, and the size of the paper is the maximum output length.

how a model writes an answer: it reads everything in the context, gives every possible next token a probability, picks one using the sampling settings, appends it to the context and repeats, until it picks a stop token or reaches the maximum output length

This loop explains several things you will see later in the series. How does this apply to Acme? Each point below is the pen rule showing up in Acme’s system:

Long answers are slow. Each output token is another trip round the loop. Part 13 separates the time to first token from the speed of the rest. A one-line answer of about 30 tokens is about 30 trips; a 300-token report is about 300, so after the first token it takes roughly ten times as long.
Streaming is possible because tokens are produced in order; the app can show them as they arrive.
The model cannot go back. Once a token is written, the next one builds on it. A wrong early token can lead a whole answer astray, which is one reason Part 5 checks numbers after the answer is written.
Thinking is the model writing extra tokens, which you do not see, before the answer. They often improve hard answers, and you pay for them as output tokens. Google’s thinking guide lists the levels each model supports; Acme’s extract step sets the level to low (Part 5) because rewriting a question needs little reasoning. For illustration: a call that reports 1,200 input tokens, 800 thinking tokens and 60 answer tokens is billed for 860 tokens at the output rate, not 60. The thinking cost more than the answer, which is why a step that only rewrites a question keeps the level low.

What is chain-of-thought, and why does thinking help?

Thinking does not make each token smarter. It gives the model more tokens, and every token is another full pass of computation. Chain-of-thought is the name for a model writing out intermediate steps before its answer; thinking is the same idea built into the model and its API.

The everyday version is a pupil asked for 17 × 24 - 19. Told to blurt the answer straight away, they may well get it wrong; given scratch paper, they rarely do. The scratch paper is the thinking tokens. The box the teacher marks is the answer you get back. And a teacher who marks only that box, right or wrong, is how the newest reasoning models are trained (below).

Where people get tripped up is picturing the model working out the answer silently and then writing the steps as an explanation. It is closer to the other way round. In the loop from the last section, each token comes from one pass through a fixed stack of layers, so the work the model can do before any one token is bounded. Writing an intermediate result puts it into the context, where every later pass can build on it:

Without room to think
  17 × 24 - 19 = ?    the next token must already be the answer: all the arithmetic
                      has to fit inside the passes that write it

With chain-of-thought (each line is tokens the model writes, then reads back)
  17 × 20 = 340
  17 × 4  = 68
  340 + 68 = 408      uses 340 and 68, which are now in the context
  408 - 19 = 389
  Answer: 389

Four easy steps, each building on numbers already written down. The paper that named the technique says the same: chain of thought “allows models to decompose multi-step problems into intermediate steps, which means that additional computation can be allocated to problems that require more reasoning steps.”

why thinking helps: without room to think, the model must produce 389 for 17 times 24 minus 19 in the passes that write the first answer token; with chain-of-thought it writes 340, 68, 408 and then 389, and each written result goes back into the context for the next pass, so more tokens buy more computation; you pay for those tokens and the answer starts only after them

Prompted chain-of-thought versus trained reasoning

There have been two generations of the idea:

Prompted chain-of-thought (2022). Wei and colleagues, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, showed that putting a few worked examples in the prompt, each with “a series of intermediate reasoning steps”, made large models much better at arithmetic, commonsense and symbolic reasoning. With just eight such examples, a 540-billion-parameter model reached state-of-the-art accuracy on the GSM8K benchmark of maths word problems. The model imitates the layout of the examples; nothing in its training rewarded checking its own steps.
Reasoning trained with reinforcement learning. Here the model is trained to think. The clearest public description is DeepSeek’s DeepSeek-R1 paper (January 2025). Its rewards were rules, not people: an accuracy reward when a maths answer matched the known result, or when code passed a suite of test cases, plus a format reward for putting the reasoning between <think> tags. Nobody showed the model how to reason, yet, the paper reports, the training “facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation”. Partway through, the authors saw an “aha moment”: a sudden rise in the model’s use of the word “wait” as it went back over its own steps.

Google does not publish how Gemini’s thinking is trained, so read R1 as the clearest public example of the idea, not as a description of Gemini.

How do you control thinking on Gemini?

You set how much the model thinks, not how. Google’s thinking guide says Gemini models “engage in dynamic thinking by default, automatically adjusting the amount of reasoning effort based on the complexity of the request”, and that you control this with thinking_level. As of November 2026 it lists low, medium and high for current models, with minimal as well on some Flash models; Gemini 3.8 Flash, for example, defaults to medium. Earlier models took a token budget, thinking_budget, instead; Google’s Gemini 3.6 migration notes say “Replace thinking_budget with thinking_level”. In the Python SDK:

config = types.GenerateContentConfig(
    thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel.LOW),
)

Three facts from the same guide decide what thinking costs you:

You pay for all of it. “When thinking is turned on, response pricing is the sum of output tokens and thinking tokens”, and pricing is based on the full thought tokens “despite only the summary being output”. That is the 800 thinking tokens in the earlier example, billed even though you never saw them.
The output cap includes thinking. max_output_tokens counts thought tokens too. If the model hits it while still thinking, you get truncated or empty output, “while still billing for any thinking tokens generated”. The guide’s advice: “To reduce cost or latency without truncating responses, lower thinking_level” instead of setting a small cap.
Thinking comes before the answer. Every thinking token delays the first word the user can read. Part 11 calls this the time to first answer token, and lists thinking as one of the three things Maria waits through before her first word appears.

When should thinking go up, and when down?

Google’s best practices give the shape: “Use minimal thinking for fact retrieval or classification”, default thinking for “comparing concepts or creative reasoning”, and maximum thinking “for advanced coding, math, or multi-step planning”.

On Acme, both sides show up:

Where thinking fails to pay. Acme’s extract step rewrites “What about Verde?” into one self-contained line. Extra thinking buys nothing there. For illustration, 800 thinking tokens spent on a 20-token rewrite would be paid on every question, and would hold up everything after it. That is why the step runs at low (Part 5).
Where thinking earns its tokens. A step like the writer turns two specialists’ results into one answer, for example why the Refill Launch lifted orders while OTIF fell. That is multi-step synthesis, the kind of task Google puts at the top of the scale.
Where neither is the answer. 17 × 24 - 19 is a teaching example. Acme never asks a model for a number; SQL computes it, and code checks it (Part 5).

Before turning thinking up, ask these in order, cheapest first:

Can code do it? Totals, dates and arithmetic go to SQL or Python, with no thinking at all.
Is it retrieval, classification or rewriting? Start at the lowest level the model supports.
Does the eval set (Part 10) pass at that level? Then keep it.
Only then raise the level one step, and measure quality, thinking tokens and time to first answer token together.

One honest gap: the thoughts you can see are summaries, and a model’s written reasoning is not a guaranteed account of how it reached the answer (Prerequisite 4 covers the evidence). Use them for debugging, not as an audit trail.

To lock it in:

TERM                   WHAT IT IS                                     EVERYDAY VERSION
chain-of-thought       steps written out before the answer            showing your working
prompted CoT (2022)    worked examples with steps in the prompt       copying a model answer's layout
RL-trained reasoning   rewarded only when the final result is right   marked on the final box only
thinking_level         how much the model may think                   how much scratch paper you get
thinking tokens        billed as output, written before the answer    thinking before you speak

What is the context window, and why does the model forget?

The bottom line: the model remembers nothing between calls; it knows only what is in the pack you send it this time, and the pack has a size limit.

Imagine a brilliant consultant with no memory at all. Before every meeting you hand them a briefing pack: their job description, the notes from earlier meetings, any reports, and today’s question. They read it all, answer, and forget everything as they leave. If last week’s notes are not in this week’s pack, last week never happened for them, and the pack can only be so many pages thick. The pack is the context window, the page limit is its size in tokens, and the forgetting is what stateless means.

How does this apply to Acme? Every call the agent makes to Gemini is one meeting, and the agent’s code assembles the pack each time.

The context window is everything the model can take into account on one call, measured in tokens: the input and the output together. Current Gemini models offer around a million tokens (Part 13), which sounds endless until you see what fills it in an agent.

what fills the context window on one model call in an agent: the system instruction, tool declarations, any loaded skill, memories, the conversation so far, tool results and the new question, plus room for the output; the model keeps none of it between calls

The most important fact on that diagram is the one people miss: the model is stateless. It remembers nothing between calls. If an agent seems to remember the last message, it is because the history was sent again, or because the API or a session store kept it and added it for you. Google’s text generation guide describes both options: the server can keep the conversation for you, or you can work “in stateless mode by managing the conversation history yourself”. ADK’s sessions (Part 8) are the second approach, done properly.

Three things follow, and each one has a part of its own:

What you send is what the model knows. Context engineering, choosing what goes in on each call, is half of Part 1.5.
More is not always better. A bigger context costs more on every call and can make answers worse, which Part 1.5 calls context rot. Part 8 keeps history small with compaction.
The stable part can be cached. Instructions and tool declarations are the same on every call, so putting them first lets caching cut their cost (Part 11).

Here is a rough estimate of how it fills up, using the rule of thumb of about 4 characters per token (the sizes are illustrative; measure real ones with count_tokens or the usage in each response):

Instruction, about 2,000 characters                                   ~500 tokens
All 151 rows of campaign_performance, about 120 characters a row      ~4,500 tokens

Call 1  instruction + question                                         ~520 tokens in
Call 2  + the model's tool call + the 151-row result                  ~5,100 tokens in
Call 3  + a second tool call and a smaller result                     ~5,600 tokens in
Call 4  next question in the same session: all of the above re-sent   ~5,700 tokens in
Call 5  + another full table result                                   ~10,300 tokens in

Nothing here is near a million tokens, yet input per call has grown about twentyfold in five calls, and you pay for all of it every time. Only the first ~500 tokens, the instruction, are identical on every call, which is the part caching can make cheaper. The large tool results and the growing history are what context engineering (Part 1.5) and compaction (Part 8) keep in check.

Which settings change the answer?

First, one word this section uses a lot: sampling. At each step the model has a probability for every possible next token, but it does not simply take the most likely one. It makes a random draw, weighted by those probabilities. Think of a bag of tickets: a token with a probability of 0.7 holds seven tickets in every ten, a token with 0.2 holds two, and the model pulls one ticket out. That weighted draw is sampling. Always taking the single most likely token instead is called greedy decoding.

Sampling is why the same prompt can give different answers, and it is also why answers read naturally rather than repeating the same safe phrase. Most of the settings below change the draw: temperature changes how the tickets are shared out, top-k and top-p take tickets for unlikely tokens out of the bag, and a seed fixes which ticket a “random” draw picks.

When you call a model you can pass settings, often called generation parameters. The Python SDK’s GenerateContentConfig (the object ADK passes through as generate_content_config) has fields for all of them. These are the ones you will meet:

the same five candidate tokens under different settings: low temperature makes the top token almost certain, temperature 1.0 spreads the chances, top-k keeps only the k most likely tokens, and top-p keeps the smallest set whose probabilities add up to p; illustrative numbers

Temperature “controls the degree of randomness in token selection”, in the words of Google’s prompt design guide. Low temperature makes the most likely token almost certain; higher temperature spreads the chances.
Top-k keeps only the k most likely tokens before picking. “A topK of 1 means the selected token is the most probable among all the tokens in the model’s vocabulary (also called greedy decoding).”
Top-p keeps tokens “from the most to least probable until the sum of their probabilities equals the topP value”.
Maximum output tokens caps the length of the reply. Hitting it cuts the answer off mid-sentence.
Stop sequences are strings that end the reply as soon as the model writes one of them. The Python SDK describes them as a “list of strings that tells the model to stop generating text if one of the strings is encountered in the response”. For example, if you ask for a one-line summary but the model sometimes adds a second paragraph of caveats, a stop sequence of a blank line ends the reply after the first paragraph:
```
config = types.GenerateContentConfig(
    stop_sequences=["\n\n"],   # stop at the first blank line
    max_output_tokens=200,       # and never past 200 tokens
)
```
It is a blunt tool: the model is not told about the stop, it is simply cut off. For a fixed shape, structured output (below) is usually the better choice.
Seed is the starting number for the random draw. A computer’s “random” numbers come from a formula, and the seed is where that formula starts, so the same seed produces the same sequence of draws. The SDK says that with a fixed seed “the model makes a best effort to provide the same response for repeated requests”; by default “a random number is used”. For example:
```
config = types.GenerateContentConfig(seed=42)
# Send the same prompt twice with this config: the replies should usually match.
# Remove the seed (or change it) and the two replies may differ.
```
The exercise at the end of this post uses the same idea: random.seed(1) makes its ten “random” picks come out the same every time you run it, which is what lets the post show you its output. A seed is useful when you want to reproduce a bad answer in a test, or compare two prompts without random noise. But it is best effort, not a promise, and it only fixes the draw for an identical request, so it is not a way to make an agent consistent.
Candidate count asks for several alternative replies to the same request (“number of response variations to return”). You pay for every one of them. Ask for three candidates on “Summarise the Refill Launch in one line” and you get three different one-liners back, and roughly three times the output tokens in the usage.
Presence penalty and frequency penalty push the model away from repeating itself. The presence penalty applies to any token that has already appeared; the frequency penalty grows each time a token repeats. Positive values make the text more varied.
Thinking level sets how much the model reasons before answering (see above).
Safety settings set thresholds for content filters. Google’s safety settings guide says categories you do not set use a default threshold “which is Off for Gemini 2.5 and later models”.

Now the part that surprises people who learned all this a year or two ago. Google’s guidance has changed:

For Gemini 3 models, Google’s Get started with Gemini 3 says: “we strongly recommend keeping the temperature parameter at its default value of 1.0”, and that lowering it “may lead to unexpected behavior, such as looping or degraded performance”.
For newer models, the prompt design guide goes further: “Gemini 4 and Gemini 3.6+ models ignore custom values for temperature, top_p, and top_k and return deprecation warning headers (and will return 400 errors in a future release)”, and asks you to remove them from your configuration.

Is this only Gemini? No. The other two big model families have made the same move, as of October 2026:

Anthropic (Claude). Anthropic’s thinking documentation says that on its current models, including Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, “non-default temperature, top_p, or top_k values return a 400 error on every request”, and that from API version 2026-10-21 a request that includes any of them “at any value returns a 400 error on every model”. The Claude Opus 5.5 migration guide says to omit them and use prompting instead, and adds a useful reminder: “If you were using temperature = 0 for determinism, note that it never guaranteed identical outputs on prior models.”
OpenAI (GPT). OpenAI’s guide to its latest models says: “When reasoning effort is not none, remove temperature, top_p, and top_logprobs.” The settings only apply when the model is not reasoning.

The pattern is the same everywhere: as models started thinking before they answer, vendors took the sampling dials away and replaced them with a thinking or effort setting. Open models you host yourself are a separate case, so check the model card and your serving software before relying on these settings.

Why would turning the dials hurt a thinking model?

The dials were designed for answers a few dozen tokens long. A thinking model writes hundreds or thousands of tokens before it answers, and on a chain that long both ends of the dial cause trouble. The vendors do not publish a full explanation, so here is what is sourced, followed by my explanation of the mechanism, labelled as such.

What the sources say:

Too low can loop. Google’s Get started with Gemini 3 says that setting temperature below 1.0 “may lead to unexpected behavior, such as looping or degraded performance, particularly in complex mathematical or reasoning tasks”. The problem is older than reasoning models: Holtzman and colleagues, The Curious Case of Neural Text Degeneration (2019), found that always choosing the most likely text “leads to text that is bland and strangely repetitive”.
Both ends hurt an open reasoning model too. DeepSeek’s R1 model card recommends a temperature “within the range of 0.5-0.7 (0.6 is recommended) to prevent endless repetitions or incoherent outputs”.

How I explain the mechanism (an explanation, not a vendor statement):

Greedy decoding can go round in circles. Recall that greedy decoding means always taking the most likely token. With it, the same context always leads to the same next step. Suppose the model writes “let me re-add the totals”, and they still match. The context now looks much as it did before the check, so the most likely next step is another check. Sampling gives the model a way out of the groove; temperature 0 takes it away.
Small risks compound over long chains. Take an illustrative 0.1% chance per token of a bad pick. A 30-token answer comes out clean 97% of the time (0.999 to the power 30). A 2,000-token chain of thought comes out clean only 14% of the time (0.999 to the power 2,000). Cut the risk to 0.01% per token and the 2,000-token chain is clean 82% of the time. Raising the temperature raises the per-token risk, and the length multiplies it. Because the model cannot go back, a bad step early in the thinking steers what follows, unless the model catches it, which is exactly the “wait” habit that reinforcement learning trains (see chain-of-thought, above).
Valid JSON never came from temperature 0. The old reason to set temperature 0 in an agent was “so the output parses”. That is now the job of structured output: the structured output guide says responses “adhere to a provided JSON Schema”, which “ensures predictable, type-safe results”, and still tells you to validate the values. For consistency in general, Google’s Gemini 3.6 notes say: “To improve determinism, define a system instruction with explicit rules.”

two ways a sampling dial can hurt a long chain of thinking: always taking the most likely token can send the model round the same loop, which Google warns about for Gemini 3; and over thousands of tokens a small chance of a bad pick per token adds up, so an illustrative 0.1 percent per token leaves a 30-token answer clean 97 percent of the time but a 2,000-token chain only 14 percent; valid JSON comes from a schema, not from temperature 0

Both sides as scenarios. Why it fails: a team copies a 2024 configuration with temperature 0 into the step that reconciles marketing and supply figures, on an earlier Gemini 3 model. By Google’s own warning, that risks looping or degraded performance on exactly the hard reasoning they wanted to be careful with; on Gemini 3.6 and later the setting is simply ignored, with a deprecation warning now and a 400 error in a future release. Why it wins: leave the sampling settings out, set the thinking level per step, give a schema where the shape matters and check the numbers in code.

So the old habit, “set temperature to 0 for agents so they are consistent”, is wrong for current Gemini, Claude and GPT models. Acme’s agents never set temperature, top-k or top-p. (The one place temperature 0 appears is the scaffold’s default judge in Part 10, which the series keeps only as a cheap signal.) It sets the thinking level per step, uses structured output where it needs a fixed shape, and gets consistency from code: fixed routes, tool results and checks. Part 10 shows the right way to handle a judge’s randomness: ask it more than once, rather than setting temperature to 0.

What else do you send besides the question?

A request is more than one line of text. Think of briefing a temp on their first morning: a job description that applies all day, the email thread so far, two examples of replies done well, a form their answer must fit, and a list of colleagues they may phone for figures. In a model request those are the system instruction, the turns, few-shot examples, structured output and function declarations. How does this apply to Acme? These are the other parts, each used later in the series:

System instruction. Standing guidance for the model, separate from the user’s message. Google’s text generation guide: “You can guide the behavior of Gemini models with system instructions.” In ADK this is the agent’s instruction (Part 1).
Roles and turns. Each message in a conversation is a turn, and each turn is labelled with who said it: its role. In the Gemini API there are two roles, user (the person, or your code) and model (the model’s earlier replies); the Python SDK says a turn’s role “must be either ‘user’ or ‘model’”. Because the model remembers nothing between calls, a follow-up question only makes sense if the earlier turns are sent with it. Here is what Maria’s third message really looks like when it reaches the model:
```
contents = [
    {"role": "user",  "parts": [{"text": "Spend on Acme Pure in Western Europe?"}]},
    {"role": "model", "parts": [{"text": "EUR 283,590."}]},
    # the new turn:
    {"role": "user",  "parts": [{"text": "How much demand did we miss there?"}]},
]
```
On its own, “How much demand did we miss there?” is meaningless: which product, which “there”? With the two earlier turns in front of it, the model can tell that “there” means Western Europe and the product is Acme Pure. Sending the history in order, every time, is what makes a model look as if it remembers. In ADK, the session (Part 8) builds this list for you.
Few-shot examples. One or more worked examples of an input and the output you want, placed in the prompt before the real input. The prompt design guide is unusually firm: “We recommend to always include few-shot examples in your prompts”, and adds that “you can remove instructions from your prompt if your examples are clear enough in showing the task at hand”. An example in the style of Acme’s extract step (Part 5), which rewrites a follow-up question into one self-contained line:
```
Rewrite the last question as one self-contained line. Keep every name exactly.

Conversation: "Spend on Acme Pure in Western Europe?" / "What about Verde?"
Rewritten: What was the spend on Verde in Western Europe?

Conversation: "OTIF for Acme Pure last week?" / "And the penalties?"
Rewritten: What were the penalties for Acme Pure last week?

Conversation: "What was the spend on Acme Pure in Western Europe?" / "And how much demand did we miss there?"
Rewritten:
```
The two worked examples show the model the pattern (keep the product, carry over the region or the week, output a single line) more clearly than a paragraph of rules would. The model then completes the third line in the same shape. Zero examples is called a zero-shot prompt; one example is one-shot; a few is few-shot.
Structured output. A JSON schema the reply must follow. The structured output guide lists “agentic workflows” among its uses. Acme’s extract step uses output_schema for the same reason (Part 5).
Function declarations. The tools the model may ask for, each with a name, a description and an input schema. The model does not run a tool; it returns a function call for your code to run, and your code sends back the result (Part 1 and Part 2).
Other media. Images, PDFs, audio and video as parts of the same request, each costing tokens.

What comes back besides text?

The bottom line: a reply is a parcel with a delivery note, and the note tells you whether the parcel is complete. A courier hands over a box, a weight for billing, and a status: “delivered”, or “partial: did not fit in the van”. Opening the box without reading the status is how you miss that half the order is missing. The box is the content, the weight is the usage, and the status is the finish reason.

A reply is a small structured object, and production code reads all of it:

The content: text, or one or more function calls.
Usage: input, output, thinking and cached token counts. Part 11’s cost report is built from these.
A finish reason: why the model stopped. The Python SDK’s FinishReason includes STOP (a natural end), MAX_TOKENS (cut off), SAFETY (blocked by a filter) and MALFORMED_FUNCTION_CALL (a tool call that does not match its schema), among others. An answer that ended with MAX_TOKENS is incomplete, however confident it sounds.

For example, suppose the output cap were set far too low, 20 tokens, for Maria’s question. The reply could stop at “The Refill Launch lifted orders by about 22%, but OTIF fell to”, with the finish reason MAX_TOKENS. The sentence reads fine up to the cut and leaves out the very figure she asked about. Code that checks for anything other than STOP catches it, and then retries with a higher cap or tells her the answer could not be completed.
Blocks. In the Gemini API’s newer interface, Google’s safety guide says that when a filter blocks the input or the reply, “the interaction fails with an HTTP 400 error”, with a code saying why.

My view: check the finish reason in code on every call. It is the cheapest guardrail there is, and a MAX_TOKENS that nobody checks is an easy way to ship half an answer that reads like a whole one.

Why do answers vary, and why can they be wrong?

The bottom line: a model gives plausible answers, not checked ones; it varies because it samples, and it can be wrong because sounding right is what it was trained for.

Ask a friend what last month’s team dinner cost. Without the receipt, they say “about 340, I think”, and tomorrow they might say 360. Both sound sure; neither is read from anything. Hand them the receipt and they read 352.80, the same every time. The friend from memory is a model with no tools; the receipt is a tool result. Where people get tripped up is thinking a lower temperature fixes this: it makes the friend say the same guess every time, not the right figure.

Because the model samples. At any temperature above 0, the same prompt can produce a different token sequence, and a small difference early on can change everything after it. And because the model predicts plausible text, not checked facts, it can produce a confident, fluent answer that is simply untrue. This is usually called a hallucination.

Here is what one looks like on Acme, the friend without the receipt. Acme’s data is private and fictional, so no model has ever seen it. Ask a model with no tools “How much did Verde spend in Southern Europe?” and it may still reply fluently, for example “Verde spent EUR 142,300 in Southern Europe” (an illustrative made-up reply). Nothing is wrong with the grammar or the tone. The figure is simply invented: it appears in no table and no tool result. In Acme’s system that answer never reaches the user, because the number check in Part 5 looks for EUR 142,300 in the tool results, does not find it and replaces the answer.

The exercise at the end shows both: at the default temperature, the model picks the right figure most of the time, and a nearby wrong figure some of the time. That is fine for wording and fatal for numbers, which is the whole reason Acme’s numbers come from tools and are checked by code (Part 5), and why the series measures behaviour with evals rather than trusting a single run (Part 10).

What is an embedding?

One more kind of model output you will meet. Think of a supermarket: nobody files products alphabetically; they go near things with a similar use, so the oat milk sits beside the soya milk even though the names share nothing but “milk”. An embedding gives every phrase a position on such a map, made of numbers, and “near” means “similar in meaning”.

An embedding is a list of numbers that represents the meaning of a piece of content, so that similar meanings get similar numbers. Google’s embeddings guide lists “semantic search, classification, and clustering” as uses. Embeddings are what make retrieval work, which Part 14 covers.

How does this apply to Acme? A toy example on supply’s vocabulary, with three numbers per phrase (real embeddings have hundreds or thousands):

"stockouts"            [0.80, 0.10, 0.30]
"missed deliveries"    [0.79, 0.12, 0.28]
"click-through rate"   [0.10, 0.90, 0.20]

The first two share almost no words but sit almost on top of each other, because they mean nearly the same thing. A search for “unmet demand” would produce a vector close to those two and far from “click-through rate”, so it finds stockout documents even though none of them uses the word “unmet”.

What will this series never ask a model to do?

This is the section the rest of the series depends on. Every rule below comes from the facts above: the model samples, it can be confidently wrong, it remembers nothing, and it follows instructions most of the time rather than all of the time.

what this series asks the model to do and what it never asks: the model reads, rewrites, writes and chooses tools; code decides routes, access, arithmetic, secrets, rules, approvals, budgets and which model runs, each in the part shown

Never let the model decide the route. Which team answers is a business rule, so code applies it (Part 5). A model choosing the route will choose differently on a different day.
Never let the model decide who may see data. Access is decided by identities and permissions in the systems themselves (Parts 3 and 7), never by an instruction.
Never trust a number the model wrote. Numbers come from tools, and code checks that every number in the answer came from a tool result (Part 5). Calculations run as code (Part 3.5).
Never put a secret or a token where the model can read it. Credentials travel in headers and metadata, read and written by code (Parts 1.5, 6 and 7).
Never enforce a rule only in the prompt. “Never write data” in an instruction is a request. Rules that must hold live in plugins and callbacks (Part 9).
Never let the model be the last check before a write. The one action that changes data waits for a person (Part 9).
Never let the model decide when to stop spending. Loops, retries and budgets are capped in code (Parts 1.5 and 11.5).
Never let a model choose its own model. Production code pins exact model IDs, and a change goes through the gate (Parts 12 and 13).
Never rely on a generation setting for consistency. Consistency comes from structure: fixed routes, structured output, tool results and checks (this part and Part 10).

All nine reduce to one test you can apply to any new step, cheapest answer first:

1. Must the result be the same every time, or be a number or a permission?   -> code
2. Can a tool fetch it from a system of record?                              -> a tool, then code checks it
3. Is it language: reading, rewriting, summarising, choosing a tool?         -> the model
4. Does it change data or spend money?                                       -> the model may propose;
                                                                                code caps it, a person approves

It is the dinner bill again: let the friend tell the story of the evening, but read the total from the receipt.

Here are three of those rules catching a model on Maria’s question, in one illustrative trace:

Rule 5: a rule in the prompt is only a request
1. The instruction says "never write data". The model calls run_query anyway with
   {"sql": "DELETE FROM campaign_performance"}.
2. A before-tool callback (code) sees DELETE and returns {"status": "error", "error": "read-only"}.
   The query never reaches the database.

Rule 3: never trust a number the model wrote
3. The supply finding says penalties were EUR 36,026.40.
4. The writer drafts "penalties were EUR 36,206.40": two digits swapped, fluent and wrong.
5. The after-model callback (code) pulls every number out of the draft, cannot find 36,206.40 in
   the findings, and replaces the draft: "I could not verify every figure. The data says: ..."

Rule 7: never let the model decide when to stop spending
6. A drafter and checker run in a LoopAgent with max_iterations=3. If the draft still fails on
   iteration 3, the loop ends because code says so, not because the model gave up.

In every step the model did the language work and code made the call. None of the three checks depends on the model following its instructions.

One honest gap: these rules make a system more predictable, not perfect. The model still writes the answer, and wording, tone and judgement calls stay probabilistic. That is what evals (Part 10) and the field guide (Parts 15 and 16) are for.

Try it: see what the sampling settings do

This needs nothing but Python. It takes five candidate next tokens with illustrative scores, turns them into probabilities at a given temperature, applies top-k and top-p, and picks ten times.

# sampler.py: what temperature, top-k and top-p do to the next token. No model, no API.
import math, random

# Illustrative scores a model might give five candidate next tokens after "Spend on Acme Pure was EUR"
logits = {" 283": 4.0, " 284": 2.5, " 280": 2.0, " about": 1.0, " unknown": 0.2}

def probabilities(logits, temperature):
    scaled = {t: s / temperature for t, s in logits.items()}
    total = sum(math.exp(v) for v in scaled.values())
    return {t: math.exp(v) / total for t, v in scaled.items()}

def candidates(probs, top_k, top_p):
    ranked = sorted(probs.items(), key=lambda kv: kv[1], reverse=True)[:top_k]   # keep the k most likely
    kept, running = [], 0.0
    for token, p in ranked:                                                    # then stop at top_p
        kept.append((token, p))
        running += p
        if running >= top_p:
            break
    return kept

random.seed(1)
for temperature, top_k, top_p in [(0.2, 5, 1.0), (1.0, 5, 1.0), (1.0, 2, 1.0), (1.0, 5, 0.6)]:
    probs = probabilities(logits, temperature)
    pool = candidates(probs, top_k, top_p)
    tokens, weights = zip(*pool)
    picks = [random.choices(tokens, weights)[0] for _ in range(10)]
    print(f"T={temperature} k={top_k} p={top_p}: pool={[t.strip() for t in tokens]} "
          f"P(283)={probs[' 283']:.2f}  ten picks: {' '.join(p.strip() for p in picks)}")

Running it prints:

T=0.2 k=5 p=1.0: pool=['283', '284', '280', 'about', 'unknown'] P(283)=1.00  ten picks: 283 283 283 283 283 283 283 283 283 283
T=1.0 k=5 p=1.0: pool=['283', '284', '280', 'about', 'unknown'] P(283)=0.70  ten picks: 284 283 284 283 283 284 283 280 280 283
T=1.0 k=2 p=1.0: pool=['283', '284'] P(283)=0.70  ten picks: 283 283 284 283 283 283 283 283 283 283
T=1.0 k=5 p=0.6: pool=['283'] P(283)=0.70  ten picks: 283 283 283 283 283 283 283 283 283 283

Look at the second line. At temperature 1.0, the default Google recommends for Gemini 3, the right figure comes out seven times in ten, and the wrong ones (284 and 280) three times. That is not a bug in the model; it is how sampling works. It is exactly why this series never lets a model be the source of a number.

Which research papers explain this behaviour?

Much of what this prerequisite says about models comes from published research, and each claim has a paper behind it. I wrote a separate series, Research papers, myth by myth, that tests one common myth per part against those papers. In one line each:

Attention Is All You Need (Google, 2017): a Transformer reads every word of the prompt at once, then writes its answer one token at a time.
Tracing the thoughts of a large language model (Anthropic, 2025): the model picks a rhyme word before writing the line, so it plans ahead even though it outputs one token at a time.
Scaling Monosemanticity and Gemma Scope (2024): researchers can now pull millions of readable features out of a model, though only a small share of what it computes.
hallucination and calibration research (2022 to 2025): made-up answers have known causes, models carry some sense of what they know, and tests that reward guessing make it worse.
Chain-of-thought prompting and reasoning faithfulness (2022, 2025): writing out steps improves answers, but the written steps often leave out what really drove the answer.
Scaling laws, Chinchilla and the emergence debate (2020 to 2023): quality improves smoothly with size, data and compute, and some sudden leaps come from how answers are scored.
Lost in the Middle and many-shot jailbreaking (2023, 2024): models miss facts in the middle of long prompts, and many examples can teach or attack a model.
RLHF, InstructGPT, sycophancy and Constitutional AI (2022, 2023): human feedback teaches what raters prefer, which includes agreeing with the user.
Sleeper Agents, alignment faking and reward hacking (2024, 2025): bad behaviour can survive safety training, so test in the setting the model will run in.
Agentic Misalignment (Anthropic, 2025): under pressure, many models chose harmful actions as agents, and a ‘do not’ line reduced it without removing it.
Red teaming and Constitutional Classifiers (2022, 2025): guardrails can cut jailbreaks sharply at a measured cost in refusals and compute.
Clio, the Economic Index and Values in the wild (2024, 2025): privacy-preserving studies show how people really use AI, mostly to help rather than replace them so far.

Parts 2, 4 and 5 matter most for this series: they are why later parts check a model’s answers with code, citations and evals instead of trusting what the model says about itself.

What did we learn?

A model is a trained function that predicts the next token. Using it does not change it, and it only knows its training data plus what you send.
A token is about four characters of English. Price, limits and latency are all measured in tokens, and images, audio and video are tokens too.
The model writes one token at a time, picking from probabilities, until it stops or hits the output limit. Thinking is extra tokens before the answer.
Training is predict, measure the loss, nudge the weights, repeated billions of times. Parameters are those weights and biases; neurons are the keys and features the chords, and because of superposition one neuron is rarely one idea.
Chain-of-thought helps because every written token is another pass of computation. Prompted chain-of-thought imitates worked examples; RL-trained reasoning is rewarded for right answers and learns to check itself. On Gemini you set thinking_level, pay for every thinking token, and wait for them before the first answer token.
The context window holds everything for one call, and the model is stateless: history must be sent again or kept for it.
Temperature, top-k and top-p shape the pick. For current Gemini models Google says to leave them at their defaults or remove them; consistency comes from structure and code.
On a long chain of thinking, temperature 0 can loop and high temperature compounds small errors; valid JSON comes from a schema, not from temperature 0.
Read the whole reply: content, usage and the finish reason.
The series never lets a model decide routes, access, numbers, secrets, rules, approvals, budgets or its own model ID.

What comes next?

Prerequisite 4: Security words for agents uses the context window from this post to explain prompt injection and the other risks every later part defends against. Then, in Production-ready agents, Part 1: What an agent really is, and the system we will build we put a model in a loop with tools and turn it into an agent, then meet the Acme system the rest of the series builds.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above.