Production-ready agents, Part 11: Deploy, observe and pay for it

In Part 10 we showed that the agents work. Unit tests check the code, evals measure the behaviour, and mutation tests confirm that the tests notice when a safety control breaks. But all of that ran on a laptop or in a test runner, and Maria in marketing cannot use a local test.

So in this part we ship the system. It answers the three questions anyone running it asks on day one:

Where does it run? Three agents on Agent Runtime, each with its own identity and its own settings, and no secrets in any file.
What did it do? A trace for every question, and a structured log entry for every tool call and every model call.
What does it cost? Measured per question that was answered correctly, not per request.

One honest gap before we start: our build has the code that measures cost, but we do not have measured cost figures yet. I show the formula, the code and a worked example with token counts that are clearly made up for illustration. I do not show a real bill.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: how to deploy, trace and cost a multi-agent system, in three minutes. The narration is a synthetic voice; the script and animation are mine.

Where are we in the architecture?

This part moves everything onto Agent Runtime and adds Secret Manager, Cloud Build, Cloud Trace and Cloud Logging. Part 1.25 walks through the whole picture, layer by layer.

where Part 11 sits in the Acme architecture: highlighted boxes are the subject of this part, solid boxes come from earlier parts, faded boxes arrive later

What do I need to know first?

Here are the terms this part uses, each with a plain explanation:

Agent Runtime: Google Cloud’s managed service for running agents in production. You give it your agent code. It builds a container (a packaged copy of your code and everything it needs), runs it, adds or removes copies as traffic changes (autoscaling) and gives you managed sessions. It is part of Gemini Enterprise Agent Platform, which Google describes as the evolution of Vertex AI. The service used to be called Vertex AI Agent Engine. The old documentation address now redirects to the Agent Runtime pages, and the API still calls a deployed agent a reasoningEngine.
Engine: one deployed agent on Agent Runtime. We have three.
Environment variable: a named value the platform hands to your running program, such as MODEL=gemini-3.5-flash. The code reads it with os.getenv("MODEL"). To change it you redeploy; you do not change the code.
Secret Manager: Google Cloud’s managed store for secrets such as passwords, API keys and certificates. IAM (Identity and Access Management) controls who can read each secret, and every read can be audited.
CI (continuous integration): every proposed change is tested automatically before it can be merged. We use Cloud Build, Google Cloud’s service that runs build steps in containers.
Trace and span: a trace is the full record of one request from start to finish. It is made of spans, and each span is one timed step, such as a model call or a tool call. Spans can sit inside other spans.
OpenTelemetry: an open standard for traces, metrics and logs. ADK writes its traces in this format, so any tool that understands the standard can read them. We send them to Cloud Trace.
Structured log: a log entry written as data, usually JSON, instead of free text. You can filter it by field, for example “every model_usage event for the writer last week”.

How is the system laid out once it is deployed?

The system is three engines, and each one is deployed on its own. The router is the only one users talk to. It calls the two specialists over A2A, as described in Part 6.

the deployed system: three Agent Runtime engines, each with its own Agent Identity and its own settings; only the router faces users; traces and logs from all three go to Cloud Trace and Cloud Logging

Each engine has its own Agent Identity, a principal (an identity that can be given permissions) created for that engine alone (Part 7), and its own environment variables. Sessions and Memory Bank are managed services that sit next to the router (Part 8). Cloud Build appears in the diagram, but it is not on the path a request takes. Its job is to test changes before they ship.

The order of deployment matters. Deploy what gets called before what does the calling. The specialists go first, because the router needs their agent card URLs as part of its settings.

A simple example first. You are opening a small shop and printing flyers. The flyer has to carry the shop’s phone number, and you only get the number once the phone line is installed. Print the flyers first and you either leave the number blank or print a guess. Install the line first, copy the real number onto the flyer, then print. The phone line is the thing that gets called, the number is its address, and the flyer is the thing that tells people where to call.

How does this apply to Acme? The specialists are the phone lines, each agent card URL is the phone number, and the router is the flyer. Here is what goes wrong in the other order:

Router first      MARKETING_AGENT_CARD_URL is still empty in env.sh
                  -> an empty value is rejected by the deploy (see the next section),
                     or, if you leave it out, the router has no address to call
Specialists first 1. deploy acb-demo-marketing (and the supply agent)
                  2. copy each agent card URL into env.sh
                  3. deploy the router, which now has both URLs

Where do settings and secrets live?

The rule is simple: the code never contains a value that changes between environments, and no file ever contains a secret. Settings are written down where everyone can see them; secrets are kept in a safe; and the best credential is one nobody has to hold.

A simple example first. A chain of cafés uses the same recipe book in every branch. Each branch pins its own notice on the kitchen wall: opening hours, which supplier, today’s specials. The combination of the safe is not on that notice; only the manager can open the safe, and the bank keeps a record of who did. Staff do not carry a key to the back door at all; their badge opens it, and a lost badge is simply switched off. The recipe book is the code, the notice is the settings, the safe is the secret store, and the badge is an identity that needs no secret.

Where people get tripped up is putting a secret into a settings file “just for now” because the file is not shared yet. Files get copied, committed and pasted into tickets; the safe is the only place a secret should ever be.

How does this apply to Acme? The same three places, one for each kind of value:

where configuration lives: settings such as ids, models and switches go in a file in git and reach the engine as environment variables; real secrets go in Secret Manager; identities need no secret at all

Settings live in env.sh.example, a file kept in version control. Its header spells out the rule: “Ids and names only. NEVER a secret.” They reach the engine through --update-env-vars.
Secrets are kept in Secret Manager and reach the engine through --secrets.
No secret at all is the best option. The engines sign in with Agent Identity, and the platform issues those credentials, so there is no key file to copy, leak or rotate (replace on a schedule).

Our agents read no secrets when they are deployed. If your agent needs one, such as a vendor API key, the configuration looks like this:

# Store the value once. Nobody needs to see it again.
printf '%s' "$VENDOR_KEY" | gcloud secrets create vendor-api-key --data-file=-

# Give it to the engine as an environment variable named VENDOR_API_KEY
agents-cli deploy --secrets "VENDOR_API_KEY=vendor-api-key"

The code still reads os.getenv("VENDOR_API_KEY"). Only the place the value comes from changes. One permission detail: when the agent uses Agent Identity, the Agent Runtime documentation says to grant roles/secretmanager.secretAccessor to the Agent Platform Service Agent, which fetches the secret during deployment.

We ran into one small trap. Agent Runtime rejects an environment variable with an empty value, and the deployment fails with a “Required field is not set” error. So env.sh.example gathers the plugin settings into one variable, PLUGIN_ENV, and leaves out any setting that is empty. This is that part of the file:

# Plugin settings (all three agents read these; the values are the code's defaults).
export ACB_KILL_SWITCH=off          # on = pause the assistant for everyone
export ACB_RATE_CAPACITY=20         # questions a person can send in a burst
export ACB_RATE_REFILL_PER_MIN=10   # questions per minute that refill the quota
export MA_TEMPLATE=""               # Model Armor template; empty = off
export MA_MODE=audit                # audit = log only, block = stop flagged prompts
# PLUGIN_ENV = the settings above in deploy format, SKIPPING EMPTY ONES.
PLUGIN_ENV=""
for _v in ACB_KILL_SWITCH ACB_RATE_CAPACITY ACB_RATE_REFILL_PER_MIN MA_TEMPLATE MA_MODE; do
  _val=$(eval "printf %s \"\${$_v}\"")
  [ -n "$_val" ] && PLUGIN_ENV="${PLUGIN_ENV:+$PLUGIN_ENV,}$_v=$_val"
done
export PLUGIN_ENV

(Shortened from env.sh.example; the real file lists a few more plugin settings.) Part 9 explains what each plugin setting does.

With the values above, the loop runs like this:

ACB_KILL_SWITCH=off          not empty -> added
ACB_RATE_CAPACITY=20         not empty -> added
ACB_RATE_REFILL_PER_MIN=10   not empty -> added
MA_TEMPLATE=""               empty     -> skipped
MA_MODE=audit                not empty -> added

PLUGIN_ENV="ACB_KILL_SWITCH=off,ACB_RATE_CAPACITY=20,ACB_RATE_REFILL_PER_MIN=10,MA_MODE=audit"

Without the skip, the deploy would send MA_TEMPLATE= and fail with “Required field is not set”. The same file is also the operations switch: set ACB_KILL_SWITCH=on, redeploy, and the assistant pauses for everyone with no code change.

How do I deploy with agents-cli?

agents-cli deploy packages the agent, builds it, and creates or updates the engine. This is the router’s deployment, simplified from the project’s deploy steps:

the router's deploy, simplified: settings come from env.sh, models are environment variables, the plugin and telemetry switches ride along, and the engine gets its own Agent Identity

The flags that matter are:

--agent-identity turns on Agent Identity. The agents-cli reference is clear about this: if you pass neither --agent-identity nor --no-agent-identity, a new agent is created without Agent Identity. Always pass it on the first deployment.
--update-env-vars sets the configuration. The models are configuration too: MODEL for the extract step and WRITER_MODEL for the writer. Changing a model means redeploying with a new value, and Part 10’s gate decides whether it ships.
--no-wait returns straight away. A deployment takes several minutes, and agents-cli deploy --status tells you when it has finished.

This design makes rollback (going back to the previous working version) easy. Because the model is a setting and not code, changing it or undoing the change is a redeploy with a different value. If a model change causes a regression, redeploy with the last MODEL and WRITER_MODEL values that worked. There is no code to revert.

A simple example first. A bakery tries a cheaper flour supplier. It changes one line on the order form, bakes a test batch and runs the same taste test it always runs. The bread is worse, so the next order goes back to the old supplier. Nobody rebuilds the oven or rewrites the recipe; the order form changed twice and the taste test made the call. The order form is env.sh, the supplier is the model, the taste test is Part 10’s eval gate, and going back to the old supplier is the rollback.

The common misconception is that rollback means digging through git history under pressure. When every value that changes between releases is a setting, rollback is the same deploy command you ran yesterday.

How does this apply to Acme? Suppose you try the writer on the cheaper model to save money (scores illustrative):

1. env.sh: WRITER_MODEL=gemini-3.5-flash          (was gemini-3.1-pro-preview)
2. deploy to staging, eval run, gate.py:
     golden_numbers   0.800 (min 0.9) FAIL
3. env.sh: WRITER_MODEL=gemini-3.1-pro-preview    (the last value that passed)
4. redeploy: the same agents-cli deploy command, --update-env-vars built from env.sh

Nothing in git changed except one line of settings, and the gate, not a person’s impression, decided that the cheaper writer was not good enough.

Which checks should run first in CI?

Run the checks that are cheap and always give the same answer first, and the slow, costly ones last. Before a pull request is merged, Cloud Build runs a fixed list of checks defined in .cloudbuild/pr_checks.yaml. By default Cloud Build runs the steps one after another, and a failed step stops the build. So the order is a design choice: the cheapest checks, which give the same result every time, run first, and the slowest run last.

A simple example first. Before a long drive you check, in this order: is there fuel in the tank (ten seconds), are the tyres inflated (a minute), then you book the car in for a full service (a day). If the tank is empty you find out in ten seconds and never wait a day at the garage to learn it. Each check runs only if the cheaper one before it passed.

How does this apply to Acme? The pipeline has the same shape: a seconds-long consistency check before the unit tests, and the unit tests before the slow mutation runs.

the CI pipeline: pull request checks run offline, cheapest first, and each step runs only if the one before passed; if a mutant survives, the build fails and nothing merges; the eval gate runs later, against deployed agents

All five steps run offline. They make no model calls and touch no data, so they are fast, cost nothing and give the same result every time:

install: installs the exact library versions from each project’s lock file (the file that pins every library version), or fails.
in sync?: each project carries a copy of the shared core, and this step fails if any copy has drifted from the others.
unit + lint: runs the unit tests, then agents-cli lint (formatting, spelling and type checks).
shared tests: runs the shared core’s tests, including the cost calculations described below.
mutants: switches off each safety control in turn and expects a test to fail (Part 10).

Here is why the order matters. Suppose someone edits factcheck.py in the router’s copy of the shared core and forgets the other two copies. Step 2, in sync?, fails within seconds. Steps 3 to 5 never start, so nobody waits for the unit tests or eight mutant runs to learn about a copy mistake.

This is one step from the real file:

- name: "gcr.io/cloud-builders/gcloud"
  id: mutation-tests
  entrypoint: /bin/bash
  dir: "acb-demo"
  args:
    - "-c"
    - |
      set -euo pipefail
      python3 tools/run_mutants.py

set -euo pipefail makes the shell stop at the first error. run_mutants.py exits with 1 if any mutant survives, so the step fails and the build fails with it.

This has already happened. I ran the mutation step on our build on 19 October 2026, and the result was 7/8 mutants killed. The survivor is token_to_marketing, which also passes the user’s Databricks token to the marketing agent. A recent change made the router do exactly that, on purpose, to support a per-user access check on the marketing side. So the fault the mutant injects changes nothing, and no test fails. The pipeline is right to stop. The team now has to decide. If the new behaviour is correct, the mutant and the specification are updated to match. If it is wrong, the change is reverted. Forcing that decision through a failed build is exactly what CI is for.

The eval gate works differently. It needs deployed agents and real model calls, so it costs money and takes longer. It lives in its own file, .cloudbuild/eval_gate.yaml, and runs after a staging deployment. One honest gap: the Cloud Build triggers are written down but not yet created, and we run the eval gate by hand.

A quick guide for where a new check belongs:

The check...                                   Put it in            Why
runs offline, same result every time, seconds  pr_checks, early     fails fast, costs nothing
runs offline but takes minutes (mutants)       pr_checks, last      only worth it once the rest pass
needs deployed agents or real model calls      eval_gate, staging   costs money, results vary a little
needs real users                               after release        cost per resolved question, below

How do I see what one question did?

A trace is a timed, nested record of every step one question took, so you can see which step was slow and which step ran twice.

A simple example first. A restaurant ticket for table 4 is stamped when the order is taken at 19:00 and closed when the bill is paid at 20:10. Inside it, the kitchen stamps each course: starter 19:05 to 19:15, main 19:15 to 19:45. Inside the main, the grill station logs 19:16 to 19:38 and the sauce station 19:30 to 19:40. If table 4 complains about the wait, the manager reads the stamps and sees that the grill took 22 minutes, not that “the kitchen was slow”. The ticket is the trace, each stamped course or station is a span, and the grill sitting inside the main is a span inside a span.

How does this apply to Acme? The ticket is one of Maria’s questions, the courses are the router’s steps, and the stations are the model and tool calls inside them. A trace shows where the time goes in a single question. This waiting time is called latency. To turn on tracing in Agent Runtime, you set three environment variables when you deploy. They are defined in env.sh.example as OTEL:

GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=EVENT_ONLY

The Agent Runtime tracing documentation describes each one. The first turns on traces, logs and metrics, but leaves out prompts and responses. The second uses the latest OpenTelemetry naming rules for generative AI (its semantic conventions). The third also logs prompts, responses and user IDs as separate log events. That third setting is a privacy decision. Only turn it on if access to those logs is restricted.

Inside the agent, the scaffold’s fast_api_app.py reads the first variable and passes otel_to_cloud=True to ADK, which sends the spans to Cloud Trace. This is what one question looks like:

the trace of one question as a waterfall: the router's agent span contains the extract model call, the call to the marketing agent with its own model and tool spans, and the writer's model call; bar lengths show the shape only, not measured times

ADK’s documentation defines the span names. invoke_agent is one agent run. generate_content is one model call, and it records the input and output token counts. execute_tool is one tool call. The waterfall view answers practical questions quickly, such as whether the writer is what slows things down, or whether the specialist called its tool twice.

Here are the spans for Maria’s “How did the Acme Pure Refill Launch perform?”, using the same illustrative token counts as the cost example below. The timings are made up too:

invoke_agent acb_demo_router                                            9.8 s
  generate_content (extract, 3.5 Flash)  3,200 in / 120 out / 300 thinking      0.9 s
  (route runs here, in code: no span with tokens)
  invoke_agent marketing (A2A)                                          5.1 s
    generate_content (3.5 Flash)         9,000 in (6,000 cached) / 150 / 400    1.4 s
    execute_tool query_marketing_measures                               1.8 s
    generate_content (3.5 Flash)         10,500 in (9,000 cached) / 350 / 200   1.6 s
  generate_content (writer, 3.1 Pro)     5,000 in / 450 out / 900 thinking      3.6 s

Reading it: the writer is the longest single model call, and most of its time is the 900 thinking tokens. Now suppose a slow day shows a second execute_tool query_marketing_measures span under marketing, with the same arguments, followed by a third generate_content. That tells you the specialist re-ran its query, so the extra seconds and tokens come from the marketing agent, not from the writer.

There are two things to keep in mind. First, the bar lengths above only show the structure; I have not published measured timings for our build. Second, the specialists are separate services. ADK says it passes trace context across process boundaries, but you should check in Trace Explorer that the specialist’s spans appear inside the router’s trace. If they don’t, match them up by timestamp and by the user ID in the logs.

Which latency number matters? TTFT, first answer token and total

For an agent, the number that matches what the user feels is the time until the first word of the answer, not the time until the model’s first token. “Latency” is really three numbers, and for an agent they can be far apart.

A simple example first. You ring a helpdesk. The line picks up after 2 seconds with a recorded “Thanks for calling, please hold”. You hear hold music while someone looks up your account. A person starts answering your actual question at 3 minutes, and the call ends at 6 minutes. The pick-up time tells you the phone system is working; the 3 minutes is what you remember. Where people get tripped up is reporting the 2 seconds on a dashboard and wondering why callers are unhappy.

How does this apply to Acme? The same three moments, with their technical names:

Total latency is the time from the user pressing Send to the last word of the answer. It is what the root span of the trace above measures. (The end of the call.)
Time to first token (TTFT) is the time until the model produces its first output token. At the model’s API, a thinking token or the first token of a tool call counts. It tells you how fast the network and the model’s reading of the input are. (The line picking up.)
Time to first answer token is the time until the first word of the answer the user can actually read appears. Thinking, tool calls and other agents’ work all happen before it. (The person starting to answer.)

Here are the three on the illustrative trace above:

one question to Acme's router on a timeline, with illustrative times: the first token of any kind leaves the first model call at about 0.3 seconds, but the first word of the answer Maria can read appears only at about 8.6 seconds, after extract, the marketing agent and the writer's thinking; the answer finishes at 9.8 seconds

The first token of any kind leaves the extract model call within a fraction of a second, but Maria cannot read anything until the writer has finished thinking, about 8.6 seconds in. Model TTFT says the system is alive; time to first answer token is what the person experiences. For a voice agent, where a caller hears silence, or for any service-level target about answers, measure the second one.

What goes into the wait before the first answer token:

Everything before the final model call: retrieval, reranking, tool calls and, in Acme’s case, a whole specialist agent over A2A.
Reading the input. Before a model writes its first token, it processes every input token: the instruction, the conversation and any retrieved text. This is often called the prefill step, and it grows with the input, which is one more reason to send 8,000 relevant tokens rather than 400,000 (Part 14).
Thinking. Thinking tokens are produced before the answer. The writer’s 900 thinking tokens are most of its time in the trace above.

And the levers, in the order I would try them:

Show progress. If the work takes eight seconds, show what is happening (“Asking the marketing team…”) as soon as it starts. Part 12.75 shows how A2A status updates do this in Gemini Enterprise. The work takes just as long, but the user can see it is happening.
Cache what repeats. Google’s context caching documentation says caching “helps reduce the cost and latency of requests to Gemini that contain repeated content”, such as a long, stable instruction.
Send less. Fewer, better chunks (Part 14) and a shorter history (compaction, Part 8) shorten the prefill.
Think less where it does not pay. A lower thinking level or budget on a step that does not need deep reasoning cuts the wait directly; check the effect with the eval set (Part 10) before keeping it.
Do less in series. Steps that do not depend on each other can run in parallel (Part 6.5), so the user waits for the slowest, not the sum.

Record both numbers. TTFT and time to first answer token can come from the timestamps of the first streamed event and the first non-thought text event; total latency comes from the trace. ADK’s evaluation also reports invocation_duration_v1 per turn (Part 10), so a change that makes answers slower shows up before it ships.

One honest note: Acme’s ADK build filters thoughts out and does not yet stream progress, so today Maria’s experienced wait is the full time to first answer token.

How do I keep logs useful and safe?

Traces tell you about timing. Logs record facts: which tool ran, with which arguments, and for whom. A useful log keeps what you will search for and hides what a thief would want, and the safe way to guarantee that is to give every log entry one way in.

A simple example first. A shop receipt prints the date, the items, the till number and your card as **** **** **** 1234. That is enough to answer “which card paid for this?” and useless to anyone who finds the receipt in a bin. The masking happens inside the till printer, so no cashier can forget to do it. The receipt is a log entry, the masked card number is a redacted value, and the till printer is the single function every entry passes through.

How does this apply to Acme? Every log entry in our build goes through one function, our till printer:

def _log(event: str, **kv: Any) -> None:
    """Structured log line: Cloud Logging parses JSON on stdout/stderr into jsonPayload."""
    _LOG.warning(json.dumps({"event": event, **redact(kv)}, default=str))

Two rules are built in. First, every entry is JSON with an event name, such as tool_call, sql_blocked, rate_limited or model_usage. Second, every value goes through redact() first. Redaction means hiding sensitive values before they are stored. redact() replaces known credential formats (Databricks tokens, Google access tokens, JSON Web Tokens, private keys) and any field with a secret-sounding name (token, password, api_key) with «redacted». Because the only way into the logs goes through redaction, a secret cannot end up in a log by accident.

Here is one call and what is stored, when the router hands Maria’s question to supply (the token value is made up):

call:    _log("tool_call", tool="query_supply_measures",
              user="maria.marketing@x", databricks_token="dapi0123example")
stored:  {"event": "tool_call", "tool": "query_supply_measures",
          "user": "maria.marketing@x", "databricks_token": "«redacted»"}

The field name contains token, so the value is hidden whatever its format. The event name, the tool and the user stay readable, which is what you need when you search the logs later.

The most important event for this part is model_usage. A plugin writes it after every model call in every agent:

the usage ledger, simplified: after every model call in every agent, one JSON log line records who asked, which question, which model and four token counts, with secrets redacted

Once these entries are in Cloud Logging, one filter brings them back:

gcloud logging read 'jsonPayload.event="model_usage"' \
  --project "$PROJECT_ID" --format=json > usage.json

Each match comes back with the ledger line in its jsonPayload. For the writer call in the cost example below, it would look like this (illustrative values; the invocation id is shortened):

{"event": "model_usage", "agent": "writer", "user": "maria.marketing@x", "invocation": "q1-router",
 "model": "gemini-3.1-pro-preview", "prompt_tokens": 5000, "cached_tokens": 0,
 "output_tokens": 450, "thinking_tokens": 900}

There is one thing to check in your own project. The Agent Runtime logging documentation says that plain stdout and stderr output is stored as text, and that the Python logger supports structured logs. Look at one entry in Logs Explorer. If it arrives as textPayload instead of jsonPayload, the filter above will return nothing, and you should use the Cloud Logging client library instead.

What does a question really cost?

Most teams start by measuring cost per request, but that number hides failures. If the agent answers badly and the user asks again, the organisation has paid twice for one answer. So we divide by something else. Divide what you spent on every question by the number of questions that were answered correctly, not by the number that were asked.

A simple example first. A bakery spends 30 pounds on ingredients and gas to bake 10 cakes, and 2 come out burnt. The cost per cake baked is 3 pounds, but the bakery can only sell 8, so the cost per cake it can sell is 30 / 8 = 3.75 pounds. The burnt cakes still cost money; they just do not count as cakes. A better oven that burns fewer cakes lowers the real cost even if each bake costs the same. (Illustrative numbers.)

How does this apply to Acme? Every question is a cake baked, and a resolved question is a cake you can sell.

A resolved question is a question the system answered correctly. In an eval run, it is an answer that passed the gate. Cost per resolved question is the total model cost of all questions in a period, divided by the number that were resolved. Failed answers stay in the total cost, because you still paid for them.

The cost of one model call comes from four token counts, and the ledger records all of them:

def call_cost(row: dict, rates: dict) -> float:
    n = lambda k: int(row.get(k) or 0)
    fresh = max(n("prompt_tokens") - n("cached_tokens"), 0)
    return (
        fresh * rates["input"]
        + n("cached_tokens") * rates["cached"]
        + (n("output_tokens") + n("thinking_tokens")) * rates["output"]
    ) / 1_000_000

Every part of this calculation comes from the public pricing page:

Thinking tokens are charged at the output rate. Gemini 3 models reason before they answer, and the Gemini pricing page lists the output price as “Text output (response and reasoning)”. So thinking tokens are added to output, not ignored.
Cached input is much cheaper. The prompt token count already includes the cached part, so the code subtracts it and prices it at the cached rate. For Gemini 3.5 Flash the list price is 1.50 US dollars per million input tokens and 0.15 for cached input.
Prices live in configuration, not in code. tools/pricing.json holds the rates for each model. I checked them on 19 October 2026 against the public list prices (global endpoint, standard tier, prompts under 200,000 tokens). Gemini 3.5 Flash: 1.50 input, 0.15 cached, 9.00 output. Gemini 3.1 Pro Preview: 2.00, 0.20 and 12.00. All figures are in US dollars per million tokens.

This is the whole file. Keys match model IDs by prefix, so gemini-3.1-pro also covers gemini-3.1-pro-preview:

{
  "gemini-3.5-flash":      {"input": 1.50, "cached": 0.15,  "output": 9.00},
  "gemini-3.1-pro":        {"input": 2.00, "cached": 0.20,  "output": 12.00},
  "gemini-3.1-flash-lite": {"input": 0.25, "cached": 0.025, "output": 1.50}
}

(The real file also carries a comment reminding you to check the rates before quoting them.)

The worked example below uses made-up token counts with a realistic spread. It covers one marketing question: one extract call, two calls inside the marketing agent and one writer call.

cost per resolved question, an illustrative example: one question costs about 5.4 US cents in model tokens, mostly fresh input and thinking; dividing the cost of all questions by the resolved ones only gives about 6.7 cents

The writer is the most expensive call, because Pro costs more per token. Thinking makes up more than a third of the cost. Cached input is the smallest part, even though it is most of the marketing agent’s prompt. If 8 of 10 questions were resolved, the real cost is 0.067 dollars per resolved question instead of 0.054. That is why this number captures quality and cost together.

This figure leaves out BigQuery bytes scanned, Databricks compute and Agent Runtime’s own compute. Add those from billing data before you quote a total cost to anyone. Part 13.5 adds the other lines and estimates a full month.

Still to do. Our build has the ledger, the report and their unit tests, but not a measured cost per resolved question. To get one, run the golden set through the deployed agents, count the questions that pass the gate, export the model_usage entries for the same time window, and run:

python3 tools/cost_report.py usage.json --resolved 42   # 42 = gate-passing answers

In production there is no gate on each answer, so you need your own definition of a resolved question. My view: start with “not replaced by the fact-check and no thumbs-down”, and say clearly which definition you used.

A toy week shows how that definition works:

questions asked                         100
replaced by the fact-check                7
thumbs-down                               5   (2 of them also replaced)
not resolved                              7 + 5 - 2 = 10
resolved                                 90

total cost        100 x 0.0537 = 5.37 USD
per resolved      5.37 / 90    = 0.0597 USD   (not 0.0537)

How do I bring the cost down without hurting quality?

Remove model work before you shrink it, and keep a saving only if the eval gate still passes. Once you can measure cost per resolved question, you can reduce it. But a cheaper agent that answers less accurately is not cheaper per resolved question. So every cost change goes through the same eval gate as any other change. In the bakery, cheaper flour that burns more cakes raises the cost per cake sold.

three cost levers, each tested through the eval gate: a smaller model for extract, routing in code at zero tokens, and context caching; a lever stays only if every gated metric holds

A smaller model for extraction. The extract step only rewrites the question as one line and may not need Flash. At the same token counts as the example, Gemini 3.1 Flash-Lite’s list price (0.25 input, 1.50 output) would bring the cost of that call down to about 0.0014 dollars from 0.0086. The change is one setting, MODEL, on the router. But the extract step feeds the routing decision, so route_correct must stay at 100 per cent. If it drops, roll back.
Routing in code. Part 5 made routing a Python function. It uses zero tokens on every question and is already in place. It is the cheapest lever, because it removes a model call completely.
Context caching. When many requests start with the same text (a shared prefix), Gemini can reuse it. Implicit caching is on by default in every Google Cloud project, and cached tokens get a 90 per cent discount. It only applies above a minimum prompt size, 4,096 tokens for most Gemini 3 models. To get more cache hits, put the content that stays the same first (instructions, skill text, tool descriptions) and the content that changes last (the question). The cached_tokens field in the ledger shows whether caching is working. Explicit caching gives you more control but also adds storage charges.

Here is the caching lever on the first marketing call of the example (9,000 prompt tokens, Flash list prices, illustrative counts):

question placed before the skill text   cached_tokens 0
  input cost = 9,000 x 1.50 / 1e6                         = 0.0135 USD
stable text first, question last        cached_tokens 6,000
  input cost = (3,000 x 1.50 + 6,000 x 0.15) / 1e6        = 0.0054 USD

Same prompt, same answer, 60 per cent less input cost for that call. The ledger’s cached_tokens field is how you see the change, so check it after you reorder a prompt.

Each lever is only a guess until the gate and the cost report both confirm it.

The litmus test I use for any step, cheapest option first:

1. Can plain code do this step?               -> do it in code, zero tokens (routing, Part 5)
2. Is most of the prompt the same each time?  -> put it first and let caching discount it
3. Can a smaller model pass the eval gate?    -> switch the MODEL setting, rerun the gate
4. Can the step think less or read less?      -> lower the thinking level, send fewer chunks (Part 14)
Only then keep the more expensive setup, and write down why next to the cost report.

Try it: run the cost report yourself

The cost report runs without any cloud resources and only needs Python 3. Save four made-up usage entries as usage.jsonl:

{"event": "model_usage", "agent": "extract", "invocation": "q1-router", "model": "gemini-3.5-flash", "prompt_tokens": 3200, "cached_tokens": 0, "output_tokens": 120, "thinking_tokens": 300}
{"event": "model_usage", "agent": "acb_demo_marketing", "invocation": "q1-mkt", "model": "gemini-3.5-flash", "prompt_tokens": 9000, "cached_tokens": 6000, "output_tokens": 150, "thinking_tokens": 400}
{"event": "model_usage", "agent": "acb_demo_marketing", "invocation": "q1-mkt", "model": "gemini-3.5-flash", "prompt_tokens": 10500, "cached_tokens": 9000, "output_tokens": 350, "thinking_tokens": 200}
{"event": "model_usage", "agent": "writer", "invocation": "q1-router", "model": "gemini-3.1-pro-preview", "prompt_tokens": 5000, "cached_tokens": 0, "output_tokens": 450, "thinking_tokens": 900}

Then run python3 tools/cost_report.py usage.jsonl --resolved 1 from the project directory. I ran it on 19 October 2026, and this is the output:

| Agent | Model | Calls | Prompt (of which cached) | Output | Thinking | USD |
|---|---|---|---|---|---|---|
| writer | gemini-3.1-pro-preview | 1 | 5,000 (0) | 450 | 900 | 0.0262 |
| acb_demo_marketing | gemini-3.5-flash | 2 | 19,500 (15,000) | 500 | 600 | 0.0189 |
| extract | gemini-3.5-flash | 1 | 3,200 (0) | 120 | 300 | 0.0086 |

Total model cost: USD 0.0537 over 2 invocations (all services)
Cost per RESOLVED question: USD 0.0537  (resolved=1)

Set every thinking_tokens value to 0 and run it again to see how much of the cost comes from thinking. The “2 invocations” figure is expected: one question creates one invocation in each service it reaches.

What can go wrong

A secret in a configuration file. Once it is committed to git, it stays in the history for good. Rotate it, then move it to Secret Manager.
An empty environment variable. Agent Runtime rejects the deployment. Leave out empty values when you build --update-env-vars.
Dividing by requests. Retries and wrong answers then look like extra traffic instead of waste. Divide by resolved questions.
Leaving out thinking tokens. They are charged at the output rate, so a cost model without them makes the agent look cheaper than it is.
Logging prompts without deciding who may read them. Content capture writes user questions and answers to the logs. Restrict access first.
Relying on old prices. Prices change. Keep them in one file and check them against the pricing page before you quote a figure.

What code did we write in this part?

Every part ends with the same map of Acme’s build: which files exist so far, and the code this part added, explained line by line. The map grows with each part.

This part shipped the three agents and made them measurable. It added a settings file that holds ids, models and plugin switches but never a secret, a CI file that runs the cheap offline checks first, a usage ledger plugin that logs four token counts after every model call, and a cost report that turns those logs into cost per resolved question using prices kept in a JSON file. Each engine is deployed with agents-cli, its own Agent Identity and the telemetry variables turned on, specialists first.

Where does it live?

router/                                             the router agent: extract, route, write
  app/agent.py                                      P5,8        workflow; extract reads memory; App with compaction and resumability
  app/app_utils/services.py                         P8          picks the session, artifact and memory services
  app/fast_api_app.py                               P8          Runner with session, artifact and memory services
  app/remote.py                                     P6,7        A2A client over mTLS; metadata carries Sam's token
  app/writer.py                                     P5,8        writer reads memories and saves the conversation to Memory Bank
  scripts/create_memory_bank.py                     P8          creates the Memory Bank and its topics, once
  tests/eval/acb_metrics.py                         P10         code metrics: route_correct, grounded, golden_numbers
  tests/eval/build_datasets.py                      P10         splits source cases into golden and held-out, seed 42
  tests/eval/calibrate.py                           P10         compares judge scores with human labels (kappa)
  tests/eval/datasets/golden_marketing.json         P10         golden cases with expected route, target and numbers
  tests/eval/eval_config.yaml                       P10         eval metrics, including the faithfulness judge
  tests/eval/gate.py                                P10         thresholds per metric; exits 1 when one fails
  tests/unit/test_router.py                         P10         unit tests; routing checked with one row per question type
marketing/                                          the marketing specialist, on BigQuery
  app/agent.py                                      P1,2,3,4    marketing agent with its skill in a SkillToolset
  app/app_utils/a2a.py                              P6          A2A endpoint and agent card (1.0 and 0.3) for marketing
  app/skills/marketing-kpis/SKILL.md                P4          marketing know-how: which tool, definitions, story checks
  app/skills/marketing-kpis/references/tables.md    P4          tables, grain, measures and the join rule, read on demand
  app/tools.py                                      P1,2        three narrow tools: measures, unserved demand, SQL fallback
  tests/unit/test_marketing.py                      P2          unit tests for the tools, with a fake context and database
supply/                                             the supply specialist, on Databricks
  app/agent.py                                      P3,7,8,9    supply agent and App with the confirmed replan tool
  app/app_utils/a2a.py                              P6          A2A endpoint and agent card (1.0 and 0.3) for supply
  app/skills/supply-catalog/SKILL.md                P4          supply know-how, given only to the supply agent
  app/tools.py                                      P3,7,9      supply tools; the replan tool asks a human first
shared/                                             code all three services share
~ adk_common.py                                     P3,6,9,11   plugins and redacted JSON logs, now with the usage ledger
  auth_manager.py                                   P7          auth manager callback: token or consent (off by default)
  engines.py                                        P3,7        run_databricks connects as the signed-in user
  factcheck.py                                      P5          ungrounded_numbers: finds figures not in the findings
  registry.py                                       P2          reads semantic.yaml: measure names, computable measures
  routing.py                                        P1          classify: picks the specialist in code, no model
  safety.py                                         P9          per-user token bucket and injection pattern scan
  semantic.yaml                                     P1,2        registry of measures, owners and published formulas
  skill_source.py                                   P4          loads a skill from disk or a pinned registry revision
  sqlguard.py                                       P3          guard_sql read-only allow-list; redact for log lines
+ tests/test_cost_report.py                         P11         unit tests for the cost maths
ops/                                                build, deploy and evaluation tooling
  .cloudbuild/eval_gate.yaml                        P10         after staging: eval run, then gate.py
~ .cloudbuild/pr_checks.yaml                        P10,11      offline PR checks, cheapest first, ending with the mutants
+ env.sh.example                                    P11         settings for every engine: ids, models, plugin switches, OTEL; no secrets
+ tools/cost_report.py                              P11         prices the ledger and reports cost per resolved question
  tools/mutants/acb_mutate.py                       P10         pytest plugin that injects each mutant's fault
+ tools/pricing.json                                P11         list prices per model, kept out of the code
  tools/run_mutants.py                              P10         runs the offline suites once per mutant

+ added in this part   ~ changed in this part   Pn = the parts that touched the file

The code, line by line

ops/env.sh.example: keeps the plugin switches in one place and drops any empty value, because Agent Runtime rejects an empty environment variable.

# Plugin settings (all three agents read these; the values are the code's defaults).
export ACB_KILL_SWITCH=off          # on = pause the assistant for everyone
export ACB_RATE_CAPACITY=20         # questions a person can send in a burst
export ACB_RATE_REFILL_PER_MIN=10   # questions per minute that refill the quota
export MA_TEMPLATE=""               # Model Armor template; empty = off
export MA_MODE=audit                # audit = log only, block = stop flagged prompts
# PLUGIN_ENV = the settings above in deploy format, SKIPPING EMPTY ONES.
PLUGIN_ENV=""
for _v in ACB_KILL_SWITCH ACB_RATE_CAPACITY ACB_RATE_REFILL_PER_MIN MA_TEMPLATE MA_MODE; do
  _val=$(eval "printf %s \"\${$_v}\"")
  [ -n "$_val" ] && PLUGIN_ENV="${PLUGIN_ENV:+$PLUGIN_ENV,}$_v=$_val"
done
export PLUGIN_ENV

export ACB_KILL_SWITCH=off: the operations switch from Part 9: set it to on and redeploy, and the assistant pauses for everyone with no code change.
export MA_TEMPLATE="": an empty value means Model Armor is off. Sent as is, this empty value would fail the deploy with “Required field is not set”.
[ -n "$_val" ] &&: adds a setting only when its value is not empty, which is how the loop skips MA_TEMPLATE.
PLUGIN_ENV="${PLUGIN_ENV:+$PLUGIN_ENV,}$_v=$_val": appends NAME=value to the list, with a comma only between entries, which is the format the deploy command’s –update-env-vars flag expects.
export PLUGIN_ENV: makes the finished list available to the deploy step, which adds it to the router’s settings along with the models and agent card URLs.

shared/adk_common.py: the usage ledger, simplified: after every model call in every agent, one redacted JSON log line records who asked, which question, which model and four token counts.

class UsageLedgerPlugin(BasePlugin):
    def __init__(self) -> None:
        super().__init__(name="usage_ledger")

    async def after_model_callback(
            self, *, callback_context, llm_response):
        u = llm_response.usage_metadata
        if u:
            _log("model_usage",
                 agent=callback_context.agent_name,
                 user=callback_context.user_id,
                 invocation=callback_context.invocation_id,
                 model=llm_response.model_version,
                 prompt_tokens=u.prompt_token_count,
                 output_tokens=u.candidates_token_count,
                 thinking_tokens=u.thoughts_token_count,
                 cached_tokens=u.cached_content_token_count)
        return None

def _log(event, **kv):
    _LOG.warning(json.dumps({"event": event, **redact(kv)}))

async def after_model_callback(: a plugin hook, so it runs after every model call in every agent of the app, without touching any agent’s own code.
invocation=callback_context.invocation_id: the invocation id groups all the model calls that served one question, so the report can add them up per question.
thinking_tokens=u.thoughts_token_count: records the model’s reasoning tokens separately, because they are billed at the output rate and are easy to forget.
cached_tokens=u.cached_content_token_count: records how much of the prompt came from the cache, which is billed at a much lower rate.
return None: None means observe only: the plugin never changes the model’s response.
**redact(kv): every value passes through redaction before it is written, so a token or password cannot reach the logs by accident.

ops/tools/cost_report.py: prices one model call from the four token counts in a ledger line.

def call_cost(row: dict, rates: dict) -> float:
    n = lambda k: int(row.get(k) or 0)
    fresh = max(n("prompt_tokens") - n("cached_tokens"), 0)
    return (
        fresh * rates["input"]
        + n("cached_tokens") * rates["cached"]
        + (n("output_tokens") + n("thinking_tokens")) * rates["output"]
    ) / 1_000_000

n = lambda k: int(row.get(k) or 0): reads one count from the ledger line and treats a missing or empty value as zero.
fresh = max(n("prompt_tokens") - n("cached_tokens"), 0): the prompt count already includes the cached part, so it is subtracted to get the tokens billed at the full input rate.
n("cached_tokens") * rates["cached"]: cached input is billed at the cached rate, 0.15 dollars per million for Gemini 3.5 Flash instead of 1.50.
(n("output_tokens") + n("thinking_tokens")) * rates["output"]: thinking tokens are added to the output, because the pricing page bills reasoning as output.
) / 1_000_000: the rates in pricing.json are dollars per million tokens, so the total is divided by a million to give dollars.

The build has the ledger, the report and their tests but not yet a measured cost per resolved question, the Cloud Build triggers are written down but not created, and on 19 October 2026 the mutants step was red at 7 of 8 killed.

What did we learn?

Latency is three numbers: model TTFT (the system is alive), time to first answer token (what the person waits for) and total latency. For agents and voice, measure the first answer token.
The system runs as three Agent Runtime engines, each with its own Agent Identity and settings, and only the router faces users.
Settings live in a file in git and reach the engine as environment variables, real secrets live in Secret Manager, and an identity is better than any secret.
CI runs the cheapest offline checks first, and a surviving mutant fails the build on purpose.
A trace shows where the time goes in one question, and structured, redacted logs record what each tool and model call did.
Cost is measured per resolved question, with thinking tokens charged at the output rate and cached input at the cached rate.
Every cost lever, such as a smaller model or context caching, stays only if the eval gate still passes.

What comes next?

The system now runs on Agent Runtime, every change is tested before it ships, and we can see what each question did and what it cost. Before real users arrive, Part 11.5: Fallbacks and budgets decides what code does when a model call fails or a question runs too long or too expensive. Then one step is left: real users. In Part 12, giving it to users, we publish the router to Gemini Enterprise, handle user consent, control which endpoints the agents may connect to with Agent Gateway, and roll out in stages: shadow mode first, then a small group, then all users.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional, and the cost example uses illustrative token counts, not measurements.