Forward deployed engineering, Part 7: Prototype in days, production in weeks

In Part 6 we chose a framework for Acme Consumer Brands: ADK, with an A2A boundary between agents owned by different teams. Now the customer wants to see something working. In my work with partners, this is the moment an engagement is most often won or lost, and it usually happens in the first week.

There are two ways to get it wrong. One is to spend six weeks building a careful system before anyone has checked that it answers the right questions. The other is to build a quick demo that impresses everyone, and then watch that demo get pushed into production with a personal key, no tests and no owner.

This part is about the path between those two mistakes. As the FDE embedded with Acme, you build a prototype in days, on purpose, and plan to throw it away. You use AI-assisted coding to move fast without lowering the bar. Then you harden the real system over a few weeks, using the checklist that the Production-ready agents series builds in detail, and hand it over so that Acme’s own team can run it.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: how a forward deployed engineer turns a throwaway prototype into a production system the customer can run, in three minutes. The narration is a synthetic voice; the script and animation are mine.

Where does this show up in an FDE job description?

Moving from prototype to production is the centre of the role in public FDE postings. From Google Cloud’s public listings, as of October 2026:

“Serve as a developer for complex AI applications, transitioning from rapid prototypes to production-grade agentic workflows (e.g., multi-agent systems, model context protocol (MCP) servers) that drive measurable return on investment (ROI).”

Forward Deployed Engineer, GenAI, Google Cloud: responsibilities

“Build high-performance evaluation pipelines and observability frameworks to ensure agentic systems meet rigorous requirements for accuracy, safety, and latency.”

Forward Deployed Engineer, GenAI, Google Cloud: responsibilities

“Co-build with customer engineering teams to instill Google-grade development best practices, ensuring long-term project success and high end-user adoption.”

Forward Deployed Engineer, GenAI, Google Cloud: responsibilities

The Staff Forward Deployed Engineer posting for Developer AI describes owning the work “end-to-end like discovery, build, roll-out, hardening, and the long tail of making it reliable”. Read together, these lines describe this whole post: prototype quickly, harden properly, and leave a team behind that can carry on.

What is a prototype, and why build one to throw away?

A prototype is a small, rough version of a system, built to answer a question quickly. Treat it as an experiment: it exists to answer one question, and its value is the answer.

There are two kinds, and it matters which one you are building:

A throwaway prototype is built to learn something and then deleted. Speed matters more than structure.
An evolutionary prototype is built to grow into the product, so it needs good structure from the start.

Production is the opposite end: the system real users depend on, run by a team that answers for it. The idea of planning a throwaway is old. In The Mythical Man-Month (1975), Fred Brooks argued that a team building a new kind of system will build a throwaway version whether it intends to or not, so it should plan for one.

My view: for agents, build a throwaway prototype almost every time. The things a prototype needs to learn, such as whether the model can answer these questions from this data, are quite different from the things production needs: identity, guardrails, tests and operations. Trying to do both at once makes the prototype slow and the production system messy.

a throwaway prototype and the production system compared: the prototype answers one question for a few people with a shared account and no guardrails, then is deleted; production serves every user as themselves, with guardrails, an eval gate, a pipeline and named owners, for years

What should the Acme prototype prove?

A prototype without a clear question turns into a demo that never ends. So before writing any code, you write a one-page prototype contract with the customer. This is the one for Acme:

PROTOTYPE CONTRACT: Acme campaign questions                 (throwaway)

Question it must answer:  Can an agent answer Maria's 20 real campaign
                          questions from the marketing tables, with the
                          right numbers?
Built with:               ADK, run in the agents-cli playground
Data:                     a read-only copy of three marketing tables,
                          eight weeks, no personal data
Skipped on purpose:       user sign-in, Databricks, the router,
                          guardrails, deployment, cost tracking
Time box:                 three days
Success:                  16 of the 20 answers correct, judged by Maria
Decision at the end:      harden it / change the use case / stop
What survives:            the 20 questions and checked answers, the tool
                          descriptions, the lessons learned
What is thrown away:      everything else

The “three marketing tables” are Acme’s campaign tables. This is the first of them, so you can picture what the prototype reads:

one of the three marketing tables, campaign_performance: one row per campaign, region and week, owned by marketing and read through BigQuery; these are the first four weeks of the Acme Pure Refill Launch in Western Europe

Prerequisite 2: The business terms and the data behind the agents explains every table and column.

Three lines in it do most of the work:

“Skipped on purpose” is written down and agreed. Nobody can later say the demo was production ready, because everyone signed a list of what it lacks.
“Success” is a number that a business user judges, and Maria’s 20 questions come from discovery (Part 2). The numbers are illustrative, but the shape is the point.
“What survives” is small and valuable. The 20 questions with checked answers become the first cases in the golden set used for evaluations (agents Part 10). The tool descriptions carry over to the real agent (agents Part 2).

A golden case is a small record: the question, plus the facts a test can check. This one comes from the agents series, simplified:

{
  "eval_case_id": "q3_delegation",
  "prompt": {"role": "user", "parts": [{"text": "What did the stockouts behind the Acme Pure Refill Launch cost us?"}]},
  "expected_route": "delegate",
  "expected_target": "supply",
  "expected_numbers": [51331]
}

Each of Maria’s checked questions becomes a case like this. Part 10 of the agents series explains every field.

The prototype ends with a decision meeting. The three outcomes are all good results. Stopping after three days because the data cannot answer the questions is far cheaper than finding out after three months.

an illustrative engagement timeline: a throwaway prototype in days, a decision to harden, change the use case or stop, then weeks of hardening, a pilot with real users and a handover to the customer's team

How does AI-assisted coding change the first days?

AI-assisted coding means using a model to help write, explain and change code. There are two levels:

A coding assistant suggests code as you type and answers questions about a codebase.
A coding agent takes a task, plans it, edits several files, runs commands and tests, and reports back for review.

Google’s own tools cover both. Gemini Code Assist is available in Standard and Enterprise editions, managed through Google Cloud. Gemini CLI is an open-source agent that runs in the terminal. Google Antigravity is an agent-first development platform with an app, an IDE, a CLI and an SDK. For agents specifically, agents-cli describes itself as a “CLI and skills for building agents on Google Cloud” that works with your coding agent: you install it with uvx google-agents-cli setup and then ask your coding agent to scaffold and build the agent.

For the Acme prototype, this turns three days of typing into three days of checking. The coding agent scaffolds the project, writes the first version of each tool and its tests, and explains parts of Acme’s existing data code that you have never seen. Your job moves from writing every line to specifying the task and reviewing what comes back.

AI-assisted coding in an engagement: you specify the task, a coding agent plans and edits, tests and checks run, you review every line and merge; sensitive code is always reviewed by hand; Google tools cover the assistant and agent levels

Three rules keep AI-assisted coding safe in someone else’s estate:

Use only the tools the customer has approved. Ask in week one (Part 3). Never paste customer data, schemas or credentials into a tool that is not approved.
You own every line. Generated code that you merge is your code. Review it as you would review a colleague’s change.
Review the sensitive parts by hand. Authentication, data access, the routing rule and anything that writes data get line-by-line review, every time.

My view: AI-assisted coding makes prototypes much cheaper, which makes the throwaway discipline more important. When a demo takes an afternoon, the temptation to ship it grows.

What separates the prototype from production?

Here is the path of one of Maria’s questions through the Acme prototype and through the production system:

the same question on two paths: in the prototype one process reads a copy of the data with a shared key and prints the answer; in production Gemini Enterprise signs Maria in, the router applies guardrails and routes in code, a specialist reads live data as Maria, and every step is traced and costed

The prototype calls one agent with a shared service account and prints the answer. The production system checks who the user is, routes the question in code, reads the data as that user (for supply data, that means Databricks as Maria), passes every guardrail, records a trace and checks the cost. The answer looks the same to Maria. Everything around it is different.

Before hardening starts, scan the prototype for shortcuts that must not survive. The Try it section at the end has a small script that does this.

What is on the hardening checklist?

Hardening means turning a working prototype into a system that is safe, tested, observable and affordable to run. Keep the checklist as a file in the repository, so that the customer’s reviewers can see progress:

HARDENING CHECKLIST: Acme router                     owner     status

Identity       each agent has its own identity, least privilege
               supply reads Databricks as the signed-in user
Data access    read-only checks, row and cost limits on queries
Guardrails     kill switch, rate limit, Model Armor, audit log
               human approval before anything that writes data
Evals          golden set and held-out set; a gate in the pipeline
Observability  one trace per question; redacted structured logs
Cost           cost per resolved question, measured and budgeted
Deployment     deployed by a pipeline, secrets in Secret Manager
Rollout        shadow mode, then a pilot group, then everyone
Handover       runbook, named owners, the customer has deployed

Each line has a part of the agents series that builds it in detail:

the hardening checklist in order: identity and data access first, then guardrails, evals, observability, cost, deployment and rollout; each card names the question it answers and the part of the agents series that builds it

Identity: who is asking, and how each agent proves who it is (agents Part 7).
Data access: callbacks that reject unsafe queries before they reach the database (agents Part 3).
Guardrails: plugins the model cannot talk its way around (agents Part 9).
Evals: tests of the agent’s behaviour that can stop a release (agents Part 10).
Observability, cost and deployment: Agent Runtime, traces, logs and the cost of a question (agents Part 11).
Rollout: publishing to Gemini Enterprise and a staged rollout (agents Part 12).

The checklist has an order. Identity and data access come first, because every other test is meaningless if the agent can read data the user is not allowed to see. Evals come before rollout, because you cannot judge a pilot without a baseline.

Why does hardening take weeks when the code takes days?

Most hardening work is not code. Someone has to approve the security design, create the identities, grant access to the data, configure the identity provider, label the evaluation cases with the business users and agree a budget. Each of those steps waits on a person at the customer who has other work. Here is an illustrative plan for Acme:

an illustrative six-week hardening plan for Acme: the security review and access requests start in the prototype week, identity and data checks come first, evals and observability run in parallel, then the pipeline, the pilot and the handover

Two habits keep the weeks from slipping:

Start the slow approvals on day one. The security review and the data access requests go in during the prototype week, not after it.
Run the work in parallel. Evals and observability do not depend on each other, so two people can build them at the same time.

How do I hand it over so it keeps running after I leave?

An FDE engagement ends. The system must not end with it. The handover is the planned transfer of the system, and of the knowledge to run it, to the customer’s team. It has three parts.

A runbook. A runbook (Google’s SRE book calls it a playbook) is a written guide to running the system: how to deploy it, how to roll it back, what the alerts mean and what to do for each known failure. The SRE book reports that recording best practices in a playbook ahead of time gives roughly a three times improvement in MTTR (mean time to repair) compared with improvising. This is the outline I use:

# Runbook: Acme router
## What it does, and what it must never do
## Who owns what (names, not teams)
## How to deploy, and how to roll back
## Dashboards and alerts, and what each alert means
## Known failures and what to do
### Answers say "I could not verify every figure" more often than usual
### The supply agent returns 401 errors
### Cost per question jumps
## How to change a prompt or a model safely (run the eval gate)
## How to report a product bug (Part 8)

Clear ownership. Every part of the system needs a named owner at the customer: the code, the prompts, the evaluation set, the cloud project, the budget and the on-call rota. Some of these belong to different teams, and that is fine, as long as each one has a name next to it.

an ownership map for the handover: every part of the system has a named owner at Acme, from the router code and pipeline to the supply agent, the evaluation set, the budget, security and the on-call rota; the names are fictional

Training by doing. Slides do not teach anyone to run a system. The pattern I use with partners is gradual release: first you build while the customer’s engineers watch, then you build together, then they build while you review, and finally they own it. The test of a good handover is simple: before you leave, someone at Acme has deployed a change, rolled it back and handled a practice incident with the runbook, without you.

gradual release during the handover: first the FDE builds while the customer's engineers watch, then they build together, then the customer builds while the FDE reviews, and finally the customer owns it; the shares are illustrative

Google’s SRE book describes a production readiness review, a structured check of a service’s reliability before an operations team takes it on. A short version of the same idea makes a good final handover meeting: walk through the checklist, the runbook and the owners together, and record anything still open with an owner and a date.

Try it: find the prototype shortcuts

This script looks for shortcuts that are fine in a throwaway prototype and must not reach production. It only needs Python 3, with no extra packages. Save it as prototype_leftovers.py:

"""Find prototype shortcuts that must not reach production. Python 3 only, no packages."""
import re
import sys
from pathlib import Path

CHECKS = [
    ("hard-coded secret", re.compile(r"(api[_-]?key|secret|token|password)\s*=\s*['\"][^'\"]{8,}['\"]", re.I)),
    ("key file in the repo", re.compile(r"service[_-]?account.*\.json|credentials\.json", re.I)),
    ("personal or shared login", re.compile(r"@(gmail|example)\.com", re.I)),
    ("print instead of logging", re.compile(r"^\s*print\(")),
    ("unbounded query", re.compile(r"select\s+\*\s+from(?![^\n]*\blimit\b)", re.I)),
    ("note to self", re.compile(r"#\s*(TODO|FIXME|HACK)\b")),
]


def scan(root: Path):
    found = []
    for path in sorted(root.rglob("*.py")):
        if ".venv" in path.parts or path.name == Path(__file__).name:
            continue
        for n, line in enumerate(path.read_text(errors="ignore").splitlines(), start=1):
            for label, pattern in CHECKS:
                if pattern.search(line):
                    found.append((label, f"{path.relative_to(root)}:{n}"))
    if not any(root.rglob("test_*.py")):
        found.append(("no tests found", str(root)))
    return found


if __name__ == "__main__":
    root = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
    results = scan(root)
    for label, where in results:
        print(f"{label:26} {where}")
    print(f"\n{len(results)} item(s) to fix or justify before hardening.")

Put it next to this fictional prototype tool, saved as notebook_agent.py:

from google.cloud import bigquery

API_KEY = "demo-key-not-a-real-one"
client = bigquery.Client.from_service_account_json("service_account.json")


def campaign_spend(brand):
    # TODO: add a row limit and a read-only check
    rows = client.query(f"SELECT * FROM marketing.spend WHERE brand = '{brand}'").result()
    print("got rows", rows.total_rows)
    return list(rows)

Run python3 prototype_leftovers.py .. When I ran it, the output was:

hard-coded secret          notebook_agent.py:3
key file in the repo       notebook_agent.py:4
note to self               notebook_agent.py:8
unbounded query            notebook_agent.py:9
print instead of logging   notebook_agent.py:10
no tests found             .

6 item(s) to fix or justify before hardening.

Then try these:

Find the one it missed. The query builds SQL with an f-string, so a brand name such as x' OR '1'='1 changes the query. Add a check for f-strings that contain SELECT, then fix the tool with a parameterised query, as agents Part 2 does.
Run it on your own prototype. Every line it prints is either a hardening task for the checklist or a decision you should write down.

The script only catches obvious shortcuts. Use the customer’s approved secret scanning and code review tools as well.

What can go wrong

The demo becomes production. A prototype that works gets shown to a director, who asks for it next week. The signed prototype contract, with its “skipped on purpose” list, is your protection.
No question, no end. A prototype without a success measure keeps growing. Agree the 20 questions and the target before you start.
Hardening starts too late. Security reviews and access requests take weeks. Start them during the prototype week.
Trusting generated code. A coding agent can write a query without a row limit or log a token. Review sensitive code by hand.
Customer data in the wrong tool. Pasting a schema or a sample into an unapproved assistant can break the customer’s data policy. Check which tools are approved first.
Handover by slide deck. If the customer’s team has never deployed or rolled back the system before you leave, the handover has not happened.
Owners by team name. “The data team owns the eval set” means nobody does. Write a name.

What I would do now

Write a prototype contract for your next engagement before writing any code, and get the business user to agree the success measure.
Start the security review and data access requests on day one.
Keep the hardening checklist in the repository, with an owner and a status on every line.
Run the leftovers script, or your own version, before hardening starts.
Book the handover rehearsal (deploy, roll back, practice incident) two weeks before you leave.

What did we learn?

A throwaway prototype exists to answer one question quickly, and planning to delete it keeps it from sliding into production.
A one-page prototype contract, with a success measure judged by a business user and a list of what was skipped on purpose, protects both sides.
AI-assisted coding turns days of typing into days of checking, as long as you use only approved tools, own every line and review sensitive code by hand.
Hardening follows a checklist in order, starting with identity and data access, because every other test is meaningless if the agent can read data the user may not see.
Hardening takes weeks because most of it waits on people, so start the slow approvals on day one and run independent work in parallel.
A handover is complete only when the customer’s team has deployed a change, rolled it back and handled a practice incident with the runbook, without you.

What comes next?

Even a well-hardened system fails in ways nobody predicted. In Part 8: Debugging in the field, we read unfamiliar code, reproduce a failure, use logs and traces to tell a customer environment problem from a product bug, and write a bug report that Product and Engineering can act on. The runbook and the named owners from this part are where that investigation starts.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional.