Production-ready agents, Prerequisite 2.75: MLOps and LLMOps: keeping models and agents right after launch

Prerequisite 2.5 trained a model and tested it on rows it had never seen. It passed. This prerequisite is about the day after: the model is in production, nobody touches the code, and a year later it is quietly wrong.

That surprises people who come from ordinary software, where a function that worked on launch day keeps working until someone edits it. Models do not behave like that, and neither do agents built on models. The discipline that keeps them right after launch is called MLOps, and for systems built on large language models, LLMOps. The two share one loop; only the things you version and test change.

Every idea starts with a simple example, a house price model or a card fraud model, and then moves to Acme. You need nothing beyond Prerequisite 2.5, and even that only for the names of a few metrics.

Where are we in the architecture?

This prerequisite does not add a box. It explains how every box stays right after launch: versions, eval gates, drift monitoring and retraining or fixing. Part 1.25 walks through the whole picture, layer by layer.

the complete Acme architecture: Maria asks in Gemini Enterprise; the router runs on Agent Runtime behind plugins and Model Armor and calls the marketing and supply agents over A2A; they reach BigQuery and Databricks through Agent Gateway; identity and secrets on the left, models, knowledge and the market watch managed agent for public research on the right, traces, logs and analytics below; above it, shoppers reach Gemini Enterprise for CX, whose conversation insights land in BigQuery, and the Legal industry edition plugs into Gemini Enterprise

What do I need to know first?

A model here is anything that turns inputs into a prediction: a house price model, a fraud score, or an LLM writing an answer. Prerequisite 2.5 explains how the first two are trained.
Features and labels. A feature is an input column (square footage); the label is the answer column the model learns to predict (the sale price).
A test set is rows held back from training, used once to measure the model before launch.
Acme’s agents, from Part 1: Maria in marketing asks questions; a router sends them to a marketing agent on BigQuery (campaign_performance) or a supply agent on Databricks (supply_service), whose owner is Sam.

What is MLOps, and why do models rot when code does not?

Code is right until someone changes it. A model is right on the day you tested it, and then the world moves away from the data it learned from. MLOps is the set of habits and pipelines that notices the move and brings the model back.

Start with two pieces of software. The first is a VAT function:

vat(100)   ->  20      today
vat(100)   ->  20      in five years, unless someone edits the rate

The second is a house price model trained on 2020 London sales. Keep it to one rule so the numbers stay small: price is about £400 per square foot.

2020   1,000 sq ft flat   predicted £400,000   sold £405,000   off by   £5,000
2025   same flat          predicted £400,000   sold £460,000   off by  £60,000

Nothing in the model changed. Nobody deployed anything. Prices moved, so the same prediction became a worse one. The numbers are illustrative, but the shape is the point: the code is identical and the error grew twelve-fold. Google’s architecture guide to MLOps puts it plainly: models “can decay in more ways than conventional software systems”, so you need to “track summary statistics of your data and monitor the online performance of your model”.

The misconception to drop first is “ship it and it’s done”. For a model, shipping is the moment it starts to age. Google’s guide calls a fully manual process, where a data scientist trains a model in a notebook and hands it over, MLOps level 0, and notes that in practice such models “often break when they are deployed in the real world” because they “fail to adapt to changes in the dynamics of the environment”. MLOps is DevOps for models: the same automation, testing and release discipline, plus one thing DevOps never needed, which is watching the data.

What does the MLOps loop look like?

Five steps, and the last one feeds the first: data pipeline, train and version, test gate, deploy, monitor. A model that skips step 5 is a model on a timer.

the MLOps loop: a data pipeline feeds training, every trained model is versioned in a registry, a test gate decides whether it ships, deployment goes through shadow and canary, and monitoring watches a world that keeps moving and triggers the next round of training

Here is the house price model going round once:

Data pipeline. A scheduled job pulls last month’s sales, cleans them, checks they still have the expected columns and types, and builds the features, such as square feet per bedroom. If the schema is wrong (a column missing, a price in pence instead of pounds), the pipeline stops rather than training on it. Google’s guide calls these data validation checks.
Train and version. The pipeline trains a candidate and records three things together: the exact data snapshot, the training code and the resulting weights. Lose any one and you cannot rebuild or explain the model. The candidate goes into a model registry, a catalogue of every trained version with its metrics, so it becomes house_price v4 next to v3.
Test gate. v4 is scored on held-out rows, the same rows v3 was scored on, and ships only if it is better. Google’s guide calls this model validation: confirming that predictive performance “is better than a certain baseline”. For a classifier such as a fraud model, better means a metric such as ROC-AUC or F1 from Prerequisite 2.5. A model that loses stops here.
Deploy. First in shadow mode: v4 prices real listings, but nobody sees its numbers, so you can compare them with v3. Then a canary: a small share of traffic, then everyone. v3 stays one step away in case you need it back.
Monitor. Compare the live inputs with the training data, and, once homes actually sell, compare real prices with predictions. When the gap crosses a threshold, trigger step 1 again.

In small numbers, step 5 looks like this:

average error on last month's sales     launch  £18,000
                                        month 9 £24,000
                                        month 14 £52,000   > alert at £30,000 -> retrain

Google’s guide names the levels by how much of this loop is automated. Level 0 is all manual. Level 1 automates the pipeline so the model is retrained on fresh data, which the guide calls continuous training (CT). Level 2 adds automated CI/CD for the pipeline itself, so a change to the training code is tested and deployed like any other software.

On Google Cloud, each step has a managed piece (names as of November 2026, after the Vertex AI products were renamed under Gemini Enterprise Agent Platform):

Data pipeline + training     Agent Platform Pipelines: runs ML pipelines written with Kubeflow
                             Pipelines or TFX, serverless, with lineage tracking
Versioning                   Model Registry on Agent Platform: every model version, aliases such
                             as default; BigQuery ML models can be registered without exporting
Test gate                    model evaluation in the pipeline, or ML.EVALUATE in BigQuery ML
Monitoring                   Model Monitoring on Agent Platform for tabular models (v1 GA on
                             endpoints, v2 in Preview per model version); ML.VALIDATE_DATA_SKEW
                             and ML.VALIDATE_DATA_DRIFT in BigQuery ML

The registry’s aliases work like a bookmark. The first version of a model automatically gets the default alias, and the docs describe aliases as working “similarly to Docker Tags or Branch references in Git”. Moving default from v3 to v4 is the release; moving it back is the rollback. Hold on to that idea: Part 13 explains the same trick on Gemini model names, where it can work against you.

What is drift? Data drift, concept drift and skew

Data drift means the inputs changed. Concept drift means the inputs look the same, but what they mean for the answer changed. A third failure, training-serving skew, is not the world moving at all: it is your own pipeline computing a feature differently in production from how it did in training.

three ways a house price model goes wrong without a code change: data drift, where the inputs move outside what it trained on; concept drift, where the same input now means a different price; and training-serving skew, where serving computes a feature differently from training

Data drift, which statisticians also call covariate shift, is the house price model meeting homes it never saw. Two ways it happens:

Inflation. In 2020 most sales in the training data were between £300,000 and £600,000, and average local salaries sat around one level. Five years on, both have moved up. The rule bigger costs more still holds, but the numbers flowing into the model sit at the edge of, or outside, what it learned from.
A new market. The company expands from London into rural Scotland, where 2,500 square foot houses with land are common. The training data had almost none. The model was never shown what such a house sells for, so it guesses; a tree-based model in particular cannot predict a price higher than the prices it was trained on.

The useful property of data drift is that you can see it without knowing the right answers. You compare the distribution of each input column today with the distribution in training. Google’s Model Monitoring calls this input feature data drift, and measures it with Jensen-Shannon divergence for numbers, and that or L-infinity distance for categories.

Concept drift is harder. Google’s ML glossary defines it as “a shift in the relationship between features and the label”. Here is an illustrative example of what that looks like. Before 2020, a flat 5 minutes from a London Tube station might have sold for about £80,000 more than an identical flat 30 minutes away; the model learned that premium from years of sales. During lockdown, many buyers stopped commuting and wanted gardens and space instead, so suppose that premium shrank to £10,000. The input column did not change at all: minutes_to_tube = 5 is still 5. What changed is what 5 means for the price. A model still adding £80,000 for it overprices every flat near a station, and no input chart would show anything odd.

Concept drift only shows up when the labels arrive: when the flats actually sell, and the sale prices come in below the predictions. That is why it is the slower one to catch.

A small terminology warning: Google’s own pages use these words a little differently. The MLOps architecture guide describes “significant changes in the data distributions” as concept drift; the ML glossary and most of the field reserve concept drift for the changed relationship and call the input change data drift. I follow the glossary. The mechanism matters more than the label: ask did the inputs move, or did their meaning move?

Training-serving skew is when the data at serving time differs from the data at training time because of how you built it. Google’s ML Crash Course separates schema skew, where training and serving data do not have the same schema, from feature skew, where “engineered data differs between training and serving”, typically because the feature code differs. In the picture above, training read floor area in square feet; a new listings feed sends square metres. A 1,000 square foot flat arrives as 93, the model reads 93 square feet, and a three-bedroom flat gets priced like a garage. No world change is needed, and no amount of retraining helps. The fix is one shared piece of feature code that both sides call.

How does the same thing look on card fraud?

Fraud models drift faster than house price models, because fraudsters change their behaviour on purpose, and the right answers arrive weeks late.

A bank’s fraud model scores each card payment from 0 to 1, and payments above a cutoff are blocked (Prerequisite 2.5 covers the cutoff). The three failures look like this:

Data drift. A new partnership brings in a wave of younger customers who pay for small things by phone. The average payment drops from £42 to £19 (illustrative). The model was never trained on this mix, and its scores shift even though nobody is committing more fraud.
Concept drift. A £2 online payment used to be almost always genuine. Then fraudsters start card testing: Stripe describes it as trying “to determine whether stolen card information is valid”, often with small payments “which cardholders are less likely to notice and report as fraudulent”. The input, £2 online, looks exactly as before; its meaning flipped.
Skew. The training data computed “payments in the last hour” from settled transactions; the live system counts attempts, including declined ones. Same column name, different number.

Now the hard part: the label for fraud is a cardholder’s dispute, and Stripe notes that card networks “typically allow cardholders to initiate disputes within 120 days of the original payment”. The model’s real recall for this week may not be known for months. So the fraud team monitors in order of speed: input drift today, the share of payments the model blocks this week, and the true fraud rate when the disputes come in.

What is LLMOps, and does an LLM drift if we never train it?

LLMOps is the same five-step loop for systems built on LLMs. You version a prompt, a model ID, tool schemas and an index instead of weights, and you gate releases on evals instead of on ROC-AUC.

The misconception here is the one I hear most: “LLMs don’t drift, because we don’t train them.” It is true that most teams never change a Gemini model’s weights. But three things around the model change all the time, and each one moves the answers:

The model changes. Providers release new versions and retire old ones. On Agent Platform, gemini-2.5-flash was retired on 20 October 2026, with gemini-3.8-flash or gemini-3.5-flash-lite listed as replacements. On the Gemini Developer API, the latest alias “will get hot-swapped with every new release”, with two weeks’ email notice. A team on an alias can get a new model without changing a line. Part 13 covers versions, aliases and retirement in full.
The data changes. Documents get rewritten, tables gain columns, and a metric’s definition gets updated by the team that owns it. The agent retrieves or queries the new version, and its answers change with it.
The users change. Launch-day questions are the ones the team expected. Three months later, people ask things nobody wrote an eval case for.

Those are the same three forces as before, in new clothes: a model change is like swapping the model in the registry without the gate, a data change is data drift, and a new kind of question is data drift in the questions themselves. And when a metric’s definition changes, the same question now has a different right answer, which is concept drift.

the same five steps for a classical model and for an LLM agent: data pipelines, versioning, a test gate, deployment and monitoring; the agent versions a prompt, a model ID, tool schemas and an index together and gates on evals instead of ROC-AUC

Walking the five steps for an LLM system:

Data pipelines become ingestion. Changed documents are parsed, chunked and turned into embeddings again; changed tables get their descriptions and metric definitions refreshed. Part 13.75 covers the preparation. Changing the embedding model is a full rebuild: Google’s embeddings docs say the embedding spaces of gemini-embedding-001 and gemini-embedding-2 are “incompatible”, and “you must re-embed all of your existing data”.
Versioning becomes a bundle. The weights are Google’s, so what you version is everything you control: the system prompt, the exact model ID, the tool schemas, the index version and the metric definitions. They are versioned together, because a prompt tuned for one model can behave differently on another, and an eval result means nothing unless you know which bundle produced it.
The test gate becomes offline evals. Instead of F1 on a test set, the release runs the eval suite from Part 10: code checks first, checks on the path the agent took, and LLM judges calibrated against people, on the golden set and the held-out set. A regression blocks the release.
Deployment is the same. Shadow first, then a small pilot group, then everyone, with the old bundle one setting away. Part 12 treats a model upgrade exactly this way.
Monitoring becomes online evals. There is no label arriving 120 days later. Instead you sample a share of live turns and score them with the same judges, and read the signals people give: thumbs-down votes and requests for a person. Every confirmed failure becomes a new offline eval case, so the same mistake cannot ship twice. Then you fix whatever caused it: the prompt, the data, a tool, or the model choice.

My view: the single most useful LLMOps habit is step 2, the bundle. Almost every “the agent got worse and nobody knows why” story I have seen ends with someone discovering that one of the five pieces changed on its own: an alias moved, a document was replaced, a tool gained a parameter. If the bundle is written down, the diff tells you what moved in a minute. If it is not, you are reading traces for a day.

How does this apply to Acme?

At Acme, the agents run the LLMOps loop and supply’s service-risk model runs the MLOps loop; both catch problems at the gate or in monitoring, before Maria or Sam notices a wrong number.

How would supply’s model stay right?

At Acme, this is how it would work. Prerequisite 2.5 trained acme_supply_ml.service_risk_v1 in BigQuery ML to predict whether OTIF (on time, in full) will fall below 95% next week for a brand and region. The loop around it:

Data. A weekly job rebuilds service_risk_weeks from supply_service.
Version. Each retrain creates a new model, service_risk_v2, v3 and so on, registered in Model Registry next to the old ones.
Gate. v2 is scored with ML.EVALUATE on the most recent weeks it never saw, and must beat v1 on recall at Sam’s threshold, measured on the same weeks.
Deploy. The supply agent’s tool reads the model name from a setting, so the switch is one change, and so is the way back.
Monitor. Every week, compare next week’s feature rows with what the model was trained on:

SELECT input, metric, threshold, value, is_anomaly
FROM ML.VALIDATE_DATA_SKEW(
  MODEL `acme_supply_ml.service_risk_v1`,
  (SELECT * FROM `acme_supply_ml.service_risk_next_week`),
  STRUCT(0.2 AS numerical_default_threshold));

ML.VALIDATE_DATA_SKEW compares serving data with the statistics BigQuery ML saved when the model was trained, so the training data is not needed again, and returns one row per feature with is_anomaly set when the difference passes the threshold (the default is 0.3; I lowered it here). ML.VALIDATE_DATA_DRIFT does the same between two sets of serving data, for example this week against last month. Suppose Acme signs a new retailer whose orders arrive in far larger batches: the orders_growth_last_week feature jumps outside anything in training and shows up as an anomaly before a single prediction is proved wrong. That is data drift caught early. The concept drift check comes later, when the real OTIF for those weeks lands and ML.EVALUATE on recent weeks shows recall falling.

How would the agents stay right?

At Acme, each release of the agents is a bundle. Written down, it would look like this (illustrative):

release            acb-2026-11-08.1
router prompt      router_v14
marketing prompt   marketing_v9
supply prompt      supply_v7
MODEL              gemini-3.5-flash
WRITER_MODEL       gemini-3.1-pro-preview
tool schemas       tools@3f2a91c
semantic.yaml      semantic@a7c04e2     otif_pct counted per order
eval report        golden: pass   held-out: pass   judges calibrated: yes

The model IDs are the ones the series pins through MODEL and WRITER_MODEL (Part 11). Now two ways the answers could drift, and what catches each.

A model version bump. Someone sets WRITER_MODEL=gemini-3.8-flash to save money. That is a new bundle, so the gate runs. Say the delegation case for Maria’s cross-team question now comes back with the right numbers but a refusal on a borderline wording that used to be answered. The gate blocks it, and the team decides between adjusting the writer prompt for the new model or staying on the old one. Without the bundle rule, the change would have shipped as “just a setting”, and the first sign would be Maria’s thumbs-down. Had the code used an alias such as gemini-flash-latest, the same change could have arrived with no setting change at all, which is why Part 15 lists it as a production failure.

A changed measure definition. Supply decides to count OTIF per order line instead of per order; Prerequisite 2 notes that companies measure it both ways. The metric view changes, semantic.yaml is updated to match, and nothing in the agents changes. Maria asks the same question as last week:

Maria: "What was Acme Pure's OTIF in Western Europe in Q3?"
last week   96.1%   (per order)
this week   93.4%   (per order line)      every tool call succeeded; nothing errored

(The numbers are illustrative.) This is concept drift for an agent: the same question, the same tables, a different right answer. Two things catch it. First, the semantic.yaml version is part of the bundle, so changing it is a release and the gate runs. A golden case that pins Q3 OTIF for Acme Pure to 96.1% now fails, and that failure is the useful part: it forces a person to decide whether the new number is intended. If it is, the team updates the expected value on purpose and the answer starts saying which definition it uses. If it is not, they roll back. Second, if the definition changed somewhere the bundle does not see, say directly in the metric view, the online evals and Maria’s feedback are the backstop, and the failure becomes a new eval case.

One short note, because the two are easy to confuse: the series builds the pinned model settings, the eval gate and the feedback log; the weekly drift job and the release manifest above show how the loop would extend them.

One honest gap: online evals tell you an answer looked wrong to a judge or a user; they do not give you ground truth the way a sold house or a fraud dispute does. A confident, well-written, wrong number can pass a judge. That is why the series keeps numbers in code and checks them in the gate (Part 10’s code metrics come first), and why the strongest drift signal for an agent is still a golden case with a pinned expected value failing on a day nobody expected it to.

Why does it fail, and why does it win?

It fails when monitoring watches the system but not the answers. It wins when every change, yours or the world’s, has to pass the same gate.

Why it fails. A team ships a house price model with uptime alerts, latency dashboards and no check on prediction error. Eighteen months later the average error has doubled, every dashboard is green, and the problem is found by a sales manager whose valuations keep coming in high. The retraining job exists, but it runs on a fixed calendar and retrains on whatever arrived, including a month where a broken feed sent prices in pence. The same failure on an agent: the code pins nothing, the provider moves an alias on a Tuesday, refusals on one kind of question rise from 1 in 50 to 1 in 8 (illustrative), and nobody sees it for three weeks because no one samples live turns.

Why it wins. A fraud team watches input drift daily, block rate weekly and confirmed fraud monthly. When card testing starts, the share of small online payments jumps on day two; the team adds a rule for the grey zone at once and retrains when the disputes confirm the pattern. On the agent side, a bundle change triggers the gate, the gate catches the regression before release instead of in Maria’s inbox, and the online evals turn the one question nobody anticipated into a permanent test.

What should you monitor first?

Monitoring everything at once is how teams end up monitoring nothing well. Start with the cheapest check that would have caught a real failure, and add the next only when the last one is running:

Are the versions pinned and written down? Model IDs, prompts, tool schemas, metric definitions, model versions in the registry. It costs nothing and turns “it got worse” into a diff.
Do schema and freshness checks run before anything trains or answers? A missing column, a unit change or a table that stopped updating is the cheapest failure to catch and the most embarrassing to miss.
Is the live input compared with training, or with last month? Data drift and skew need no labels: ML.VALIDATE_DATA_SKEW, Model Monitoring, or for an agent, the mix of questions and tools called per day.
Are live outputs sampled and scored? Online evals and user feedback for an agent; the distribution of predictions for a classical model.
Do real outcomes come back and get compared with predictions? Sale prices, disputes, actual OTIF. This is the only check that sees concept drift, and it is the slowest.
Only then, automate retraining. Continuous training on top of checks 1 to 5 is powerful. Continuous training without them automates the damage: the pipeline retrains on the broken feed and ships it.

Lock it in

TERM                    WHAT IT IS                                   PICTURE IT AS
MLOps                   DevOps for models: pipelines, versions,      servicing a car on a schedule,
                        gates, rollout, monitoring, retraining       not only on the day you buy it
LLMOps                  the same loop for LLM systems                the same servicing, different parts
Model registry          every trained version, with aliases          a library catalogue with editions
Data drift              the inputs moved                             rural Scotland in a London model
Concept drift           same input, different right answer           the Tube premium in lockdown
Training-serving skew   your own pipeline disagrees with itself      square feet in, square metres out
Test gate / evals       a new version ships only if it beats         the driving test, retaken
                        the current one on the same cases            every time
Online evals            scoring sampled live answers                 a mystery shopper
The bundle              prompt + model ID + tools + index +          one recipe card, not five notes
                        definitions, versioned together

If I had to say it in four sentences: shipping a model is day one, not the finish line. A classical model needs a loop of data pipelines, a registry, a gate, a careful rollout and monitoring for data drift, concept drift and skew, with retraining when the scores fall. An LLM system needs exactly the same loop, except that instead of versioning weights and gating on ROC-AUC, you version the prompt, the model ID, the tool schemas and the index together and gate every release on offline evals, then watch it with online ones. And the model, the data and the users will all change whether or not you ever train anything.

What can go wrong

Monitoring the servers, not the answers. Uptime and latency can be perfect while every prediction is wrong. Watch error, or for an agent, eval scores and feedback.
Retraining on bad data automatically. Continuous training without schema checks and a gate retrains on the broken feed and ships it. Validate the data before training, and gate the model before deploying.
Versioning the model but not the bundle. You pinned gemini-3.5-flash but edited the prompt in place, so last month’s eval result describes a system that no longer exists.
Treating a definition change as a data refresh. A changed metric definition changes the right answer. Put definitions in the bundle so the change runs the gate.
Waiting for labels that come late. Fraud disputes can take months, and some agent answers never get a label at all. Watch inputs and sampled outputs in the meantime.
An alias in production. latest can move to a new model with two weeks’ notice by email. Pin exact IDs and change them on purpose, through the gate.
Forgetting the retirement date. A pinned model still retires. Put the date in the calendar, and run the gate on the replacement well before it.
Changing the embedding model without re-embedding. Old and new vectors are not comparable, so retrieval quietly degrades. Rebuild the index as part of the same release.

What did we learn?

Code stays right until someone changes it; a model decays as the world moves away from its training data. MLOps is DevOps for models, plus watching the data.
The MLOps loop has five steps: data pipeline, train and version, test gate, deploy through shadow and canary, and monitor, which triggers the next round. Google calls automated retraining continuous training.
A model registry keeps every version with its data, code and metrics; moving an alias such as default is the release and moving it back is the rollback.
Data drift means the inputs moved and can be seen without labels; concept drift means the same input now has a different right answer and only shows when labels arrive; training-serving skew is your own pipeline computing a feature two ways.
Fraud drifts faster than house prices, because fraudsters adapt on purpose and disputes can arrive up to 120 days later, so monitor inputs first and outcomes when they come.
LLMOps is the same loop: ingestion and re-embedding, a versioned bundle of prompt, model ID, tool schemas, index and definitions, offline evals as the gate, shadow and pilot rollout, and online evals that turn failures into new cases.
LLMs drift even if you never train them, because the model, the data and the users all change.
At Acme, a model version bump and a changed OTIF definition are both bundle changes that run the gate; a pinned golden case failing is the signal that the right answer moved.
Monitor cheapest first: pinned versions, schema and freshness, input drift, sampled outputs, real outcomes, and only then automated retraining.

What comes next?

Prerequisite 3: How a model works opens up the LLM itself: how training sets its weights, what a token is, why the context window forgets, and which jobs this series never gives to a model. Keep this post’s loop in mind as you read it; Part 10 builds the gate, Part 11 the observability behind it, and Part 15 is the field guide for when something drifts anyway.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above, as of November 2026; the house price, fraud and Acme numbers are illustrative. Acme Consumer Brands and its data are fictional.