Production-ready agents: what this series covers

Start here: better agents, done simply, the whole series in about 15 minutes. If you are new to agents, watch this first. It builds the whole system one box at a time, with one simple example, and each chapter points to the part that goes deeper.

Video: better production-ready agents, done simply: the whole series in about fifteen minutes. The narration is a synthetic voice; the script and animation are mine.

You can now build an agent that answers a question in a few hours. Building one that a company can trust with real users, real data and real money is much harder, and most agents never get past the demo.

This series is about closing that gap. Starting from 3 short prerequisites, I build one agent system on Google Cloud from the ground up and explain every design decision as I go, then show how to choose its models, decide where its knowledge lives and fix it when it breaks. By the end, you should be able to read the code of a production agent, know why each piece is there, and keep it running.

If you prefer to watch first, here is the whole series in a short animated explainer.

Video: what the production-ready agents series covers, in three minutes (recorded when the series had its first 12 parts). The narration is a synthetic voice; the script and animation are mine.

What does the finished system look like?

Every part of the series adds one or more boxes to this picture, and each part opens with the same diagram, its own boxes highlighted. Part 1.25 walks through the whole picture, layer by layer.

the complete Acme architecture: Maria asks in Gemini Enterprise; the router runs on Agent Runtime behind plugins and Model Armor and calls the marketing and supply agents over A2A; they reach BigQuery and Databricks through Agent Gateway; identity and secrets on the left, models, knowledge and the market watch managed agent for public research on the right, traces, logs and analytics below; above it, shoppers reach Gemini Enterprise for CX, whose conversation insights land in BigQuery, and the Legal industry edition plugs into Gemini Enterprise

Is there a video version?

Yes. Each part gets a short animated explainer of its core idea. Here are the first two.

Part 1: what a model, a tool and an agent are, in three minutes. Read Part 1.

Part 2: how to give an agent one clear job and narrow tools, in under three minutes. The narration in both videos is a synthetic voice; the scripts and animation are mine.

Who is this series for?

Engineers who have never built an agent. Part 1 starts from what a model is, and I explain every term the first time it appears.
Engineers who have built a prototype agent and want to know what is missing before real users touch it.
Architects and technical leads who need a clear picture of what “production ready” means for agents, so that they can plan and scope the work.

You don’t need to be fluent in Python. Where I show code, I show only the lines that matter and explain them one line at a time. If you have never read Python, start with the Python prerequisite. It covers exactly the Python this series uses, with real agent code as the examples. Then read Prerequisite 2, which explains the business terms, the tables and the real data behind every agent in the series.

What are we going to build?

Three agents that answer one question across two data platforms, without copying data from one to the other. The example is deliberately ordinary, because the hard part is not the question.

An everyday example first. A corner shop runs a two-for-one offer on bread. The owner looks at the till and says the offer was a hit: sales doubled. The person who stocks the shelves says it was a disaster: the shelves were empty by noon and regulars went elsewhere. Both are right. The full story only appears when you put the till roll next to the stock book.

How does this apply to Acme? One example runs through the whole series. A company called Acme Consumer Brands runs an advertising campaign. The marketing team says it was a success. The supply team says it was a failure. Both are right. The advertising data lives in BigQuery (Google Cloud’s data warehouse), the delivery data lives in Databricks (a separate data platform), and nobody has ever put the two side by side.

the two tables behind every agent, three real rows each: campaign_performance has one row per campaign, region and week, owned by marketing in BigQuery; supply_service has one row per brand, region and week, owned by supply in Databricks

These are three real rows from each side. Prerequisite 2 explains every column and the finding they reveal when you put them together.

Over the series we build three agents that answer questions across both sources without copying data from one to the other:

a marketing agent that queries BigQuery,
a supply agent that queries Databricks as the user who asked the question,
a router agent, the one users talk to in Gemini Enterprise. It sends each question to the right specialist and writes one combined answer.

Here is the finished system answering one question end to end. The marketing agent returns media spend of EUR 283,590 and a CTR of 0.95% from campaign_performance. The supply agent returns OTIF of 87.4% and 15,011 unmet units from supply_service. The router combines them into one answer: orders rose about 22% during the Refill Launch, but OTIF fell to 87.4% and penalties reached EUR 36,026. The rows above already hint at it: in the week of 4 May, marketing spent EUR 36,317.41 on the campaign while supply’s OTIF for the same brand and region was 89.18%, below the 95% target. No table was copied from one platform to the other to get there.

the system this series builds: a user asks in Gemini Enterprise, the router agent sends the question to the marketing and supply agents, each reads its own data, and one checked answer comes back

Acme Consumer Brands is fictional and its data is made up. The architecture and the code are real, and they run on Google Cloud.

What does “production ready” mean?

A prototype agent and a production agent use the same model. The difference is everything you build around the model: who the agent acts as, what it is never allowed to do, what it remembers between conversations, how you check that it behaves correctly, and how you run it and pay for it.

An everyday example first. A good home cook can make a great curry for six friends. Opening a restaurant with the same cook and the same recipe is a different job: the kitchen needs a hygiene rating, allergens must be recorded for every dish, the till must charge the right price, a manager must know who is on shift, and someone must notice when a dish starts coming back. None of that changes the recipe. All of it decides whether strangers can safely eat there.

The cook and the recipe        ->  the model and its prompt
Who is on shift, who may cook  ->  identity: who the agent acts as
Allergen rules, locked stores  ->  guardrails: what it may never do
The order book and regulars    ->  sessions and memory
Tasting before a dish goes out ->  evals
The till and the books         ->  running and paying for it

A common mistake is to treat the gap as a model problem and look for a cleverer model. The cook was never the problem.

a demo agent and a production agent use the same model; the difference is everything built around it, and each part of this series adds one of those layers

Each part of the series adds one of these layers to the same system.

Here are two of those layers on the same question, “What did the Acme Pure stockouts in Western Europe cost us?”. How does this apply to Acme? Suppose a quick demo version of the agent:

Demo
1. The agent connects to the data with the developer's own login.
2. It can read any table the developer can, including the confidential penalty data,
   so whoever asks gets the figure, whatever their own access.
3. The instruction says "read only", but if the model sends a DELETE, nothing stops it.

Production (what this series builds)
1. The router sends the penalty question to the supply agent (delegation).
2. The supply agent queries Databricks as the person who asked, so Unity Catalog applies
   that person's own grants. Someone without access gets no figure.
3. A callback checks every query before it runs. A DELETE never reaches the database:
   the tool returns {"status": "error", "error": "read-only"} instead.

The model is identical in both. The second version is safe because of code and permissions around the model, not because of anything the instruction says. In restaurant terms, step 2 checks who is on shift before anyone opens the store, and step 3 is the lock on the store: the cook never had to be better. Part 3 builds the read-only callback, and Part 7 covers querying Databricks as the user.

What is the one design rule?

Models read and write language. Code makes the decisions.

A model is very good at making sense of a vague question and writing a clear answer. It is not reliable at deciding which user may see which figure, or which system to call. Those decisions have to be deterministic (the same input always gives the same output), so I write them in code. This rule applies in every part, starting with Part 1, where the same rule closes the section “Who decides: the model or my code?”.

An everyday example first. At an airport check-in desk, the person behind the desk does the language: they understand “I’ve got one big case and a guitar, is that all right?”, explain the options and answer questions. The decisions that must be the same for everyone are not left to their mood: the scale weighs the case, the airline’s rule sets the weight limit, and the system decides whether there is a fee. Ask on Monday or Friday, in a good mood or a bad one, and the same case gets the same answer.

The person at the desk          ->  the model: understands the question, writes the reply
The scale and the weight rule   ->  code: the same input always gives the same output
"Is that all right?"            ->  a vague question in natural language
"That is 2 kg over, so EUR 30"  ->  an answer written from a figure code produced
                                    (the fee is illustrative)

How does this apply to Acme? Here is the rule on one question. Maria, in marketing, asks: “For Acme Pure in Western Europe, what were our media spend, our OTIF and our penalties?”

1. Model (language): turns the question into a list of measures:
   ["media_spend_eur", "otif_pct", "service_penalty_eur"]
2. Code (decision): looks each measure up in the shared registry, semantic.yaml:
   media_spend_eur      owned by marketing                      -> local: marketing's own table
   otif_pct             owned by supply, formula published      -> federated: read supply's shared table
   service_penalty_eur  owned by supply, formula not published  -> delegated: ask the supply agent
3. The agents fetch the figures: EUR 283,590, 87.4%, EUR 36,026.
4. Model (language): writes the answer from those figures.

The model never chose a route. Run the question a hundred times and code sends the penalties to the supply agent every time. If a model chose the routes, one run in many could send penalties to the marketing agent, which has no access to them.

Where people get tripped up is putting a rule like that in the instruction, for example “only ask the supply agent about penalties”, and trusting the model to follow it every time. An instruction is a request; code is a guarantee. The registry lookup in step 2 is the scale at the check-in desk.

How is the series organised?

the whole series on one page: three prerequisites, then five stages: build the agents (Parts 1 to 4 with 1.25, 1.5, 1.75, 1.9 and 3.5), connect them (Parts 5 to 8 with 6.5), protect and prove them (Parts 9 and 10), ship them (Parts 11, 11.5, 11.75, 12, 12.5, 12.75 and 12.9) and master them (Parts 13 to 17, with 13.5)

The series starts with prerequisites, then five stages. Part numbers follow reading order: decimal parts such as 1.25 or 11.5 were added later and slot in where they belong. This is a living series, and I add parts as the platform and my own work move on, so expect more. The prerequisites come first and need no agent knowledge; skip them if you already know Python and the business terms.

If you are not sure where to start, ask yourself these questions in order and stop at the first yes:

1. Never read Python?                    ->  the Python prerequisite
2. Not sure what a token or a context    ->  Prerequisite 3: How a model works
   window is?
3. Never built an agent?                 ->  Part 1, then read the stages in order
4. Built a prototype, want the picture?  ->  Part 1.25: the complete architecture
5. Running an agent that misbehaves?     ->  only then jump to Parts 15 and 16, the field guide

Stage 1: Build the agents

Prerequisite 0: Before you build: what to have ready. Nine things to settle before writing agent code: a problem worth an agent, a measure of success, ready data, sorted governance, an identity plan, a risk decision, an eval set, an owner with a budget, and the people who will use it.

Prerequisite 1: The Python you need to read agent code. Data structures, functions, type hints, docstrings, imports, classes, exception handling and async, each shown with real agent code. If you already write Python, you can skip it.

Prerequisite 2: The business terms and the data behind the agents. The business question, the marketing and supply terms (CTR, CPM, OTIF, penalties, cost to serve) with real numbers, the four tables and who owns them, the join that reveals the finding, and how the system decides whether to federate or delegate each number.

Prerequisite 2.5: Traditional AI and generative AI. When a small predictive model beats an LLM: supervised and unsupervised learning, features and labels, train, validation and test splits, regression, decision trees and boosted trees, the accuracy trap, precision, recall, F1, ROC-AUC and the cutoff, then neural networks and the four families that lead to the Transformer.

Prerequisite 2.75: MLOps and LLMOps. Why models rot when code does not: the loop of data pipelines, versioning, eval gates, careful rollout, drift monitoring and retraining; data drift, concept drift and training-serving skew; and the same discipline for agents, where the prompt, model ID, tools and index are versioned together.

Prerequisite 3: How a model works. What a token is and why it sets price, limits and speed; how a model writes one token at a time; the context window and why the model remembers nothing; temperature, top-k, top-p and the other settings (and why Google now says to leave them alone); finish reasons; and the jobs this series never gives to a model.

Prerequisite 4: Security words for agents. Why a model cannot tell instructions from data, direct and indirect prompt injection, why an instruction to keep a secret is not a control, the lethal trifecta, and the OWASP Top 10 for LLM applications, each mapped to the part that defends against it.

Prerequisite 5: Compliance for AI systems. For architects: controller and processor, provider and deployer, GDPR for prompts, memory and logs, the EU AI Act’s risk levels and dates, sector rules, ISO/IEC 42001 and the NIST AI RMF, provider questions, and how each obligation maps to a control in the series.

Part 1: What an agent really is, and the system we will build. The basics: what a model is, what a tool is and what makes something an agent. Then a first look at the Agent, App and Runner classes in Google’s Agent Development Kit (ADK), the full architecture of the system and the design rule above.

Part 1.25: The complete architecture. The finished system on one page, drawn with official product icons and explained area by area: who asks, what runs the agents, what keeps them safe, where the models and data sit, how a change ships and how you see what happened. Then one question traced from end to end. Every part opens with the same picture, its own boxes highlighted.

Part 1.5: The harness, the loop and the context. Three disciplines that sit underneath every other part: harness engineering (the code that wraps the model, runs its tools and stores state), loop engineering (how an agent plans, acts, observes and stops) and context engineering (what goes into the model’s window on every call). It also maps all ten agent-engineering disciplines (tools, memory, orchestration, guardrails, evals, human-in-the-loop design, observability and these three) to the parts that cover them.

Part 1.75: Agent frameworks, and where ADK fits. What an agent framework is, why ADK is one framework among several (and open source), and an in-depth comparison of ADK, LangGraph, CrewAI, the OpenAI Agents SDK, Microsoft Agent Framework and the Claude Agent SDK across the same production concerns, with the same Acme step in each.

Part 1.9: How agents connect: MCP, A2A and UTCP. The three kinds of connection behind every arrow in the architecture, explained once: MCP from an agent to its tools (with elicitation and the stateless 2026 spec), A2A from one agent to another, and direct calls to existing APIs with UTCP or OpenAPI tools, with a test for choosing between them.

Part 2: Your first agent: one job, narrow tools. We build the marketing agent. You will see why a tool’s description works as its contract, why a small set of narrow tools is safer than letting the model write any query it likes, and how to run the agent on your own machine with agents-cli.

Part 2.5: Inside ToolContext. The object ADK passes to every tool, which the model never sees. How ADK fills it, what ToolContext | None = None means, session state versus user:, app: and temp: state, and how a tool reaches Memory Bank, with the usage patterns that keep credentials safe.

Part 3: Letting an agent touch data safely. Callbacks (functions that run before and after a tool call). One callback rejects any query that is not read-only before it reaches the database, and every query has row and cost limits.

Part 3.5: Letting an agent write and run code safely. Why agents write code, why running model-written code is dangerous, what a sandbox is, and the ADK code executors from Gemini’s built-in tool to an Agent Runtime sandbox or your own GKE cluster, with the Acme analysis step that never holds credentials.

Part 4: Skills: teaching an agent how. A tool defines what an agent can do. A skill defines how to do a task, packaged as a folder with a SKILL.md file in it. This part covers how skills are loaded only when needed and how to publish them to a registry.

Stage 2: Connect them

Part 5: The router: let code choose the path. ADK graph workflows. The model tidies up the question, plain Python picks the route, and two specialists can run at the same time. A second model writes the answer, and code checks every figure in it.

Part 6: Agents talking to agents. The A2A (Agent2Agent) protocol: how the router finds and calls the specialists, and how the identity of the user who asked travels with each request without ever going into a prompt.

Part 6.5: Multi-agent topologies beyond a router. Eight ways to arrange agents (sequential pipeline, parallel fan-out, hierarchical supervisor, agent as a tool, loop and critic, handoff, shared state and a mesh over A2A), the ADK building block for each, when each is worth its cost, and what goes wrong as topologies grow.

Part 7: Who is asking? Identity for agents. Every call carries two identities: the agent’s and the user’s. This part covers Agent Identity on Google Cloud, querying Databricks as the user, keeping tokens out of what the model sees, and a real bug we hit with certificate-bound tokens.

Part 8: Memory: what an agent should remember. Sessions, state and Memory Bank: what the agent should remember about a user from one conversation to the next, and what it must never store.

Stage 3: Protect and prove

Part 9: Guardrails that cannot be talked around. Plugins (code that applies a policy to every agent in the app): a kill switch, rate limiting, Model Armor against prompt injection, and a human approval step before the one action that changes data.

Part 10: Proving it works: evals that gate releases. Unit tests for the code and evaluations (tests of the agent’s behaviour) for everything else: golden questions, checks written in code, LLM judges that we trust only after measuring how accurate they are, and mutation tests that switch off each safety control on purpose to prove the tests notice.

Stage 4: Ship

Part 11: Deploy, observe and pay for it. Deploying to Agent Runtime on Google Cloud, a CI pipeline that runs the cheapest checks first, tracing and logging, and the cost figure that matters most: cost per question answered.

Part 11.5: Fallbacks and budgets. What code does when a model call fails: a ladder of deterministic fallbacks, from bounded retries to an honest templated decline. And how to enforce token, cost, model-call and time budgets while a turn runs, not afterwards in a report.

Part 11.75: What Gemini Enterprise is. The product we publish to, explained: the Gemini agent, connectors and actions, skills, the two meanings of “plugin”, and sandbox-based agents, with a map of what Microsoft, OpenAI and Anthropic call the same pieces.

Part 12: Giving it to users. Publishing the router to Gemini Enterprise, user consent, controlling where the agent can send traffic with Agent Gateway, and a staged rollout: shadow mode first, then a pilot group, then general availability.

Part 12.5: Front doors and runtimes. Gemini Enterprise compared with the other front doors (your own app, Google Chat, Slack and other agents over A2A), where else an agent can run (Cloud Run, GKE, any container), and how to write agent code that does not care which front door called it or where it runs.

Stage 5: Master it

Part 12.75: How agents show more than text. The three open standards for agent UI: A2UI (a JSON description of cards and forms that the app draws itself), AG-UI (the event stream between an app and an agent) and MCP Apps (a tool’s own HTML, run in a sandbox), how Gemini Enterprise and ADK support them, and where each fits at Acme.

Part 12.9: Long-running agents: work that carries on after the tab closes. Agents that start on a schedule or an event, keep their state, pause for a person’s approval and finish hours later: the four building blocks, the 3 a.m. token problem, keeping credentials out of the sandbox, and whether the agent should act as a person or as itself.

Part 13: Choosing and running models for agents. The model families you will meet (Gemini, Claude and GPT, plus the open-weight Gemma, Llama, Mistral, Qwen and DeepSeek), what matters for an agent beyond the leaderboards, how to compare candidates on your own evals, a different model per step, and how versions, aliases and retirement dates work.

Part 13.5: What does an agent cost to own? The total cost of ownership, line by line (tokens, runtime, state, data, gateway, front door and people), a worked monthly estimate for Acme at list prices, and what each architecture choice does to the bill: ADK or LangGraph, MCP, UTCP or A2A, Agent Runtime or Databricks Agent Bricks, and the UI options.

Part 13.75: Preparing data for agents: structured and unstructured. For tables: curate, describe, define each metric once, teach with examples, check quality and freshness, permission at the source. For documents: inventory, parse with the layout, clean, chunk by structure, enrich and keep fresh. Then test both before the agent sees them.

Part 14: Knowledge: RAG, long context, fine-tuning and the infrastructure behind them. Where an agent’s knowledge should live (in the weights, in the context or behind a tool), whether RAG is still relevant, what Google Cloud offers for retrieval, when fine-tuning is really worth it, and what each option takes to run.

Part 15: What breaks in production: answers, loops, cost and model changes. The first half of a field guide: twenty failure patterns, each with the symptom, the likely causes, how to confirm which it is from traces, logs and evals, the fix, and how to stop it coming back.

Part 16: What breaks in production: identity, agents, safety and rollout. The second half of the field guide, then an incident triage flow, a runbook template and a recap of the whole series.

Part 17: Designing an agent system from scratch. A framework for assessing any new agent system (frame and discovery, flow, fence, feedback, finances and rollout), applied first to Acme and then to five real-world scenarios: a central AI platform, a factory technician assistant, 10 million marketing emails, a contact-centre bot and long document review.

How is each part laid out?

Every part follows the same pattern:

The key ideas, each term in bold with a plain explanation.
The architecture, as a diagram. Many diagrams are animated so you can watch a request move through the system.
The code, only the lines that matter, with notes on what each line does and why it is there.
What can go wrong, taken from real problems we hit during the build.
Sources: links to the public documentation behind every fact.

What do I need to follow along?

A Google Cloud project, if you want to run the code yourself.
Python 3.11 or later.
For the Databricks parts, a Databricks workspace on Google Cloud. You can still follow those parts without one.

Some products in the later parts are in Preview on Google Cloud (available to try, but not yet generally available). I label each one clearly, with the date I checked its status.

What did we learn?

The series builds one agent system on Google Cloud, from the first definition to real users, and explains every design decision on the way.
The running example is Acme Consumer Brands: marketing data in BigQuery, supply data in Databricks, and three agents that answer across both without copying data.
A production agent uses the same model as a demo agent; what differs is everything built around the model.
The one design rule is that models read and write language, and code makes the decisions.
The parts fall into five stages: build the agents, connect them, protect and prove them, ship them to users, then master them.

What comes next?

Start with Prerequisite 0: Before you build. If Python is new to you, continue with Production-ready agents, Prerequisite: The Python you need to read agent code. Then read Prerequisite 2: The business terms and the data behind the agents, which explains the two tables above and every number the agents quote, then Prerequisite 2.5: Traditional AI and generative AI and Prerequisite 2.75: MLOps and LLMOps, and Prerequisite 3: How a model works, then Prerequisite 4: Security words for agents and Prerequisite 5: Compliance for AI systems. After that, Part 1: What an agent really is, and the system we will build starts from the beginning.

Sources

Opinions are my own. Every fact in this series comes from public documentation, linked in each part.