Sheet 02 / 05 — Blogs← Archive

Production-ready agents, Prerequisite 2.5: Traditional AI and generative AI: from regression and trees to neural networks

Not every AI question needs a large language model. This prerequisite explains traditional machine learning from the ground up and where generative AI fits beside it: supervised, unsupervised and reinforcement learning; features, labels, one-hot encoding and ratios; train, validation and test sets, time-ordered splits and leakage; linear and logistic regression, decision trees and gradient-boosted trees, regularisation and cross-validation; the accuracy trap, precision, recall, F1, ROC-AUC and the cutoff threshold; neural networks, activation functions and the four families (MLP, CNN, RNN and Transformer); which model wins which kind of data; and how an agent calls a BigQuery ML model through a narrow tool instead of asking an LLM to guess a probability.

8 November 2026· 49 min read · ~1 h 11 min listen· Agents · Machine learning · BigQuery ML · Fundamentals

Most of this series is about generative AI: a model that reads a question and writes an answer. But a lot of the questions people bring to an agent are not writing jobs at all. Which of next week’s brand and region combinations will miss the delivery target? Which leads will buy? What will we sell next month? Those are scoring jobs, and a different, older kind of machine learning has done them well for decades.

I see teams reach for an LLM first and ask it for a probability. It will happily give one, and the number will sound right. It is still the wrong tool. This prerequisite explains the right one, traditional machine learning, from the ground up, and then shows where generative AI comes from: the same building blocks, scaled up into neural networks.

It covers the three ways machines learn, what a feature and a label are, why you split data three ways and by time, how regression, decision trees and gradient-boosted trees work, why accuracy lies on rare events, and how precision, recall, F1, ROC-AUC and the cutoff threshold fit together. Then it opens up a neural network, walks through the four families that matter (the last one, the Transformer, is what every LLM in this series is built on), and ends with which model wins which kind of data. Finally I show how Acme would train a small model in BigQuery ML and let its supply agent call it through a narrow tool.

You need no maths beyond fractions. Every idea starts with a simple example (a house price, a driving test, a hospital screening 1,000 patients) and then moves to Acme.

Where are we in the architecture?

This prerequisite does not add a box. It explains the other kind of model a tool behind any box can call: predictive models that score rows, and how they differ from the Gemini models on the right. Part 1.25 walks through the whole picture, layer by layer.

the complete Acme architecture: Maria asks in Gemini Enterprise; the router runs on Agent Runtime behind plugins and Model Armor and calls the marketing and supply agents over A2A; they reach BigQuery and Databricks through Agent Gateway; identity and secrets on the left, models, knowledge and the market watch managed agent for public research on the right, traces, logs and analytics below; above it, shoppers reach Gemini Enterprise for CX, whose conversation insights land in BigQuery, and the Legal industry edition plugs into Gemini Enterprise

Traditional AI or generative AI: which one does the job?

The bottom line: generative AI writes; traditional, predictive ML scores. When the job is to put a number or a probability on a row of a table, train a small model on your own rows and let the LLM explain the result.

Start with an estate agent selling a house. Two jobs come up. One is writing the listing: a warm paragraph about the light in the kitchen. The other is valuing the house: £580,000, based on what similar houses nearby sold for. You would happily let a gifted writer draft the listing. You would not let them pick the price from a feeling; you would look at the sales records. The listing is generative AI: it produces text. The valuation is predictive ML (also called classical or tabular machine learning): a model learns from past rows, each with a known answer, and predicts the answer for a new row.

The misconception I hear most often is “Gemini is smarter than a little regression model, so ask Gemini for the probability.” Here is why that fails, and why the small model wins:

Why the LLM fails at scoring. Ask an LLM “What is the chance that Acme Pure misses 95% OTIF in Western Europe next week?” and it writes a plausible number, say 70%. That number did not come from Acme’s history; the model has never seen supply_service. Ask again and you may get 65%. Score 600 brand and region rows and you pay for 600 prompts and still cannot say how often a “70%” turns out right. Prerequisite 3 lists numbers among the jobs this series never gives to a model, and this is the same rule.
Why the small model wins. A logistic regression or a gradient-boosted tree model trained on two years of supply_service rows scores all 600 rows in one SQL query, returns the same probability every time for the same inputs, and comes with a measured track record: on weeks it never saw, how often its warnings were right and how many real misses it caught. That track record is what the rest of this post teaches you to read.

So where does the agent fit? The predictive model scores; the LLM orchestrates and explains. The agent decides that a score is needed, calls the model through a tool, and turns the result into a sentence a person can act on.

the predictive ML pipeline: raw rows become features and a label, the rows are split by time into train, validation and test, a model is trained, it is judged on the test weeks with precision, recall, F1 and ROC-AUC, and a threshold turns each probability into an action; the LLM sits outside the pipeline and calls the last step through a tool

How do machines learn? Supervised, unsupervised and reinforcement learning

The bottom line: supervised learning learns from examples that come with the right answer; unsupervised learning finds groups and oddities in data that has no answers at all. Google’s introduction to ML sorts machine learning into four kinds: supervised, unsupervised, reinforcement learning and generative AI.

Picture two piles of old house sales. In the first, every sale has its final price written on it. You can study the pile and learn how area, bedrooms and location relate to price, then estimate the price of a house that has not sold. That is supervised learning: every training example has a label, the right answer, and the model learns to predict it. When the label is a number (a price) the job is called regression; when it is a category (sold within 30 days, yes or no) it is classification.

In the second pile nobody wrote anything down. You can still spread the sales out and notice that they fall into natural groups: small flats near the station, large family houses in the suburbs, cottages in the villages. Nobody told you those groups exist; you found them. That is unsupervised learning: no labels, just structure. Two jobs come up again and again:

Clustering puts similar rows into groups. The most widely used method is k-means: you choose how many groups (k), and it finds k centre points and assigns each row to the nearest.
Anomaly detection flags rows that do not look like the rest, such as a sale price far outside anything similar.

Here is k-means on six shoppers, described by visits per month and average basket, with k = 2:

Shoppers: A (2, £20)  B (3, £25)  C (2, £22)  D (10, £80)  E (12, £90)  F (11, £85)

Step 1  Pick two starting centres: A and D.
Step 2  Assign each shopper to the nearer centre:  {A, B, C} and {D, E, F}.
Step 3  Move each centre to the average of its group: (2.33, £22.33) and (11, £85).
Step 4  Assign again. Nobody changes group, so k-means stops.

Result: two segments. The algorithm does not know what they mean; you name them
"occasional, small baskets" and "frequent, big baskets".

That last line matters: the crash course points out that an unsupervised approach does not know what the clusters mean, and it is up to you to define them.

The third kind, reinforcement learning, learns by trying actions and receiving rewards or penalties; it trains robots and game players such as AlphaGo, and it is how today’s reasoning models learn to think, which Prerequisite 3 explains. The fourth, generative AI, is the rest of this series.

How would Acme use each kind?

At Acme, this is how it would work. Sam’s question, will this brand and region miss 95% OTIF next week?, is supervised classification: two years of weeks where the answer is known. Maria has an unsupervised job too. She could cluster brand and region combinations in campaign_performance by click-through rate and cost per thousand impressions, and find, say, a “cheap reach, few clicks” group and an “expensive but clicked” group that she never defined in advance. BigQuery ML has a KMEANS model type for that, and ML.DETECT_ANOMALIES can use a k-means model to flag a week that looks unlike the others. The rest of this post follows Sam’s supervised model, because that is where most business scoring lives.

What is a feature here, and how is it different from the features inside an LLM?

The bottom line: in predictive ML, a feature is an input column and the label is the answer column you want to predict. It is a different thing from the features researchers find inside an LLM.

The word feature has two meanings in AI, and this series uses both. In Prerequisite 3 and Prerequisite 4, a feature is a concept a large model represents internally, such as the Golden Gate Bridge feature that interpretability researchers can watch and turn up or down. Nobody types those in; they are discovered inside the weights. In predictive ML, a feature is far plainer: a column in your spreadsheet that you choose and compute.

Take one house in a table of past sales:

property_type  area_sqft  bedrooms  date_built   sold_price
Detached           2,000         2  1996-05-12   £580,000

A feature (often written X) is an input column: property_type, area_sqft, bedrooms.
The label (often written y) is the answer column you want to predict: sold_price for a number, or sold_within_30_days = 1 or 0 for a yes or no.
A row with a known label is a training example. The model learns how the features relate to the label, then predicts the label for houses that have not sold yet.

What is feature engineering?

Feature engineering is building better input columns from raw data, using what you know about the business. Three moves cover most of it.

One-hot encoding. Models do arithmetic, so they cannot multiply the word “Detached”. One-hot encoding turns a text category into one 0 or 1 column per value: is_detached = 1, is_flat = 0, is_semi = 0. Keep real numbers as numbers, though: area_sqft = 2500 should stay 2500, so the model knows 4,000 is bigger than 2,500. One-hot encoding is for categories, not quantities.
Time aggregates. A raw date such as date_built = 1996-05-12 means nothing to an equation. Turn it into something countable: house_age = 2026 - 1996 = 30 years, days_on_market = 45, or price_drops_last_30_days = 2 (a count of events over a time window, which is what aggregate means here).
Ratios. House A has 2,000 sq ft and 2 bedrooms: large, generous rooms. House B has 2,000 sq ft and 8 bedrooms: small, cramped ones. Add sqft_per_bedroom = area_sqft / bedrooms and House A gets 1,000 while House B gets 250. One ratio tells the model more about luxury than either column alone.

What would Acme’s features look like?

At Acme, this is how it would work. Sam in supply wants an early warning: will this brand, in this region, miss the 95% OTIF target next week? (OTIF, on time in full, is delivered units over ordered units; Prerequisite 2 explains it and the 95% contract line.) One row per brand, region and week, built from supply_service and the campaign_calendar view, with the same three moves:

Predicting the week of 4 May 2026, using only what was known by the end of 27 April:

brand        Acme Pure
region       Western Europe     -> one-hot: region_western_europe = 1, region_southern_europe = 0
otif_last_week             95.49   (27 April: 13,691 delivered / 14,338 ordered)
orders_growth_last_week    1.084   (ratio: 14,338 / 13,223 ordered the week before)
campaign_starts_next_week  1       (the Refill Launch starts on 4 May, from campaign_calendar)
weeks_below_95_last_8      0       (time aggregate)

label: otif_below_95 = 1           (4 May came in at 89.18%)

Every number above comes from the demo data in Prerequisite 2. Notice what is not a feature: units_delivered for the week of 4 May. On 27 April nobody knows it yet, and it is most of the answer. Putting it in would be leakage, which the next section is about.

Why split the data three ways?

The bottom line: train on one part, tune on a second, and keep a third locked away for one honest final score. A set you keep checking against stops being unseen.

Think of learning to drive. Your driving lessons are on the same streets every week. Then your instructor gives you a mock test on a new route; you fail the parallel parking, practise, and take mock tests until you pass. Finally you sit the real test on a route you have never driven, once, with no retakes. Map it across:

The lessons are the training set (often about 70% of the rows): the model adjusts its weights or tree splits on these.
The mock tests are the validation set (often about 15%): you use them to choose hyperparameters, the settings you pick before training rather than the model learning them, such as how deep a tree may grow.
The real test is the test set (the last 15% or so): scored once, at the end.

Where people get tripped up is asking “validation and test are both unseen, so why do I need two?” After your tenth mock test on the same route, you have tuned yourself to that route. It is no longer unseen. The same happens to a model: every time you check a validation score and change a setting, your choices leak a little information about that set into the model. Google’s Machine Learning Crash Course puts it directly: validation and test sets “wear out” with repeated use. The test set stays honest only if you do not tune against it. (The 70, 15, 15 split is a common starting point, not a rule.)

This is the same idea as the golden set and the held-out set in Part 10: the golden set is the mock test you work through every day, the held-out set is the real test you open once at release.

Why must time-ordered data be split by time?

The bottom line: if your rows have dates, split by date: train on the past and test on the future, because that is how the model will be used.

The trap is splitting rows at random. Suppose you have a year of sales and shuffle them into train and test. Your training set now holds 22 November and your test set holds 21 November. If a viral promotion ran that week, the model learned about it in training and then “predicts” it in test. It has seen the future, and its test score is too good to be true. Nothing looks broken; the model just fails the first time it meets a week it truly has not seen. Rows that are close in time are alike (this week’s orders look like last week’s, which statisticians call autocorrelation), so a random split leaks neighbours across the line.

random split versus time-ordered split on twelve weeks: a random split puts week 8 in training and week 7 in test, so the model has already seen the future it is tested on; a time-ordered split trains on weeks 1 to 8, validates on weeks 9 and 10 and tests once on weeks 11 and 12

How does this apply to Acme? The Refill Launch ran for seven weeks from 4 May 2026, and every one of those weeks missed 95%. Shuffle at random and some launch weeks land in training while their neighbours land in test; the model scores beautifully on test because it has already met that launch. Split by time instead: train on the oldest weeks, validate on the next, test on the newest, and no launch week sits on both sides.

There are two kinds of leakage, and the Acme row above shows both:

Leakage through the split: a random shuffle of dated rows, as in the diagram.
Leakage through a feature: a column that would not exist at prediction time, such as the target week’s units_delivered. The test is simple: could I have known this value at the moment I would make the prediction? If not, it is not a feature.

One practical warning for BigQuery ML users. Its default DATA_SPLIT_METHOD, AUTO_SPLIT, randomises the rows (with 500 to 50,000 rows it holds back 20% at random as evaluation data). For dated rows choose SEQ, which sorts by a column you name and holds back the last rows, or CUSTOM, where your own column says which rows go where. SEQ sorts on a NUMERIC, BIGNUMERIC, STRING or TIMESTAMP column, so a DATE column such as week_start needs a TIMESTAMP(week_start) first.

What are regression, decision trees, random forests and gradient-boosted trees?

The bottom line: regression adds up weighted features; a decision tree asks a chain of yes or no questions; a random forest lets many trees vote; gradient-boosted trees are a relay of small trees, each one trained to fix the error the trees before it left behind.

Which model families are there?

Before the details, a map. Most models you will meet belong to one of five families, and they differ in how they turn features into an answer:

Linear models add up weighted features: linear regression for a number, logistic regression for a probability. The next subsection works through both.
Tree-based models ask yes or no questions: one decision tree, a random forest of many trees that vote, or gradient-boosted trees that work as a relay. This is where XGBoost and LightGBM live, and it is the family that usually wins on business tables.
Distance and margin models classify by geometry. k-nearest neighbours finds the k training rows closest to a new row and predicts from their labels: to price a house, take the five most similar sales and average them. A support vector machine looks for the boundary with the widest gap to the nearest training points of each class; those nearest points are the support vectors. scikit-learn’s guides to nearest neighbours and support vector machines define both.
Neural networks stack many weighted sums in layers; the neural network section below covers them.
Clustering, the unsupervised family from earlier, needs no labels: k-means, or DBSCAN, which treats clusters as dense areas separated by sparse ones, so they can be any shape (scikit-learn: clustering).

machine learning model families at a glance: linear models predict a number or a probability from a weighted sum; tree-based models, from one decision tree to random forests and gradient-boosted trees, predict a number or a class and suit business tables; distance and margin models such as k-nearest neighbours and support vector machines classify by closeness or by the widest gap; neural networks handle images, audio and text; clustering such as k-means and DBSCAN finds groups without labels

What are linear and logistic regression?

Linear regression predicts a number as a weighted sum of the features plus a constant, the bias: price = w1 x area + w2 x bedrooms + b. Training finds the weights. Suppose it learns w1 = £150 per sq ft, w2 = £20,000 per bedroom and b = £240,000. For the house from earlier:

price = 150 x 2,000  +  20,000 x 2  +  240,000
      = 300,000      +  40,000      +  240,000   = £580,000

Each weight is readable: one more square foot adds £150, whatever else is true. That is the strength of the model, and also its limit.

Logistic regression is the yes or no version. It computes the same kind of weighted sum, called z, and then squashes it into a probability between 0 and 1 with the sigmoid (also called the logistic function), 1 / (1 + e^-z). A z of 0 gives 0.5; a z of 1.2 gives 1 / (1 + 0.30) = 0.77; a z of -2 gives 0.12. For Sam’s row, a large positive weight on campaign_starts_next_week and a negative one on otif_last_week would push z up in a launch week that starts from a weak position. Training these weights is the same loop that trains an LLM, described in Prerequisite 3: predict, measure the loss, nudge the weights, repeat. The model is just far smaller.

What is inside a decision tree, and how does it choose the first question?

A straight-line formula struggles with real life. Being near a station might add a lot to a house price in a city and nothing in a remote village; the effect depends on other columns. A decision tree handles this by learning questions instead of weights.

Take an illustrative loan decision. Google’s Decision Forests course describes the parts: every non-leaf node holds a condition (also called a split or a test), and every leaf holds a prediction.

The root node is the first condition, here credit score below 700?
Internal nodes, or condition nodes, ask further questions of the rows that reached them: income below £40,000? on one side, debt above £20,000? on the other.
Leaves give the answer: approve or deny. For a regression tree, a leaf holds a number, such as a house price.

anatomy of a decision tree for a loan, illustrative: the root node asks whether the credit score is below 700; internal nodes ask whether income is below £40,000 or debt is above £20,000; the four leaves give approve or deny; an applicant with a score of 680 and an income of £35,000 follows yes, yes and is denied

How does training pick credit score as the root rather than income? It tries candidate conditions and keeps the one that best separates the labels, then repeats on each side: a greedy, step-by-step search, as the course’s page on growing decision trees explains. For classification the usual scores are information gain (based on entropy) and Gini impurity; for regression it is usually the mean squared error of each side, which amounts to picking the split that makes the numbers on each side least spread out (often called variance reduction).

Gini impurity is simple enough to compute by hand. For a group where a share p of rows is “approve”, it is 1 - (p² + (1 - p)²), as in Google’s ML glossary: 0 for a pure group, 0.5 for a 50:50 mix. With ten toy applicants, five approved and five denied:

Start:                 5 approve, 5 deny          Gini = 1 - (0.5² + 0.5²)       = 0.50

Split on credit score < 700:
  yes side (4 rows):   1 approve, 3 deny          Gini = 1 - (0.25² + 0.75²)     = 0.375
  no side  (6 rows):   4 approve, 2 deny          Gini = 1 - (0.667² + 0.333²)   = 0.444
  weighted:            0.4 x 0.375 + 0.6 x 0.444                                = 0.42

Split on income < £40,000:
  yes side (5 rows):   2 approve, 3 deny          Gini = 0.48
  no side  (5 rows):   3 approve, 2 deny          Gini = 0.48
  weighted:                                                                     = 0.48

Credit score lowers impurity more (0.50 to 0.42, against 0.48), so it becomes the root.

Each leaf is one answer. A single deep tree, though, can grow a leaf for almost every training row. That is overfitting: learning the training rows by heart, noise and all, instead of the pattern behind them. The two fixes below both use many trees.

How do random forests and gradient boosting combine trees?

The bottom line: a random forest grows many deep trees independently on shuffled data and lets them vote; gradient boosting grows shallow trees one after another, each fixing the last one’s error.

Picture a shop predicting how many cartons of milk store 42 will sell in the next hour (toy numbers), with features such as current_stock and sales_last_1hr.

Random forest (bagging). Grow, say, 100 trees at once, each slightly different. Three trees might predict 40, 44 and 41 cartons; the forest answers their average, 41.7. For a yes or no question such as will it run out?, the trees vote, or (as scikit-learn implements it) their probabilities are averaged. The Decision Forests course’s page on random forests explains the two sources of difference between the trees. Bagging (bootstrap aggregating) trains each tree on a random sample of the training rows, drawn with replacement, so some rows appear twice and others not at all. Attribute sampling tests only a random subset of the features at each node, instead of looking for the best condition over all of them. The trees are grown deep and not pruned; each one overfits, but their errors are only weakly related, so they largely cancel out.
Gradient boosting. Grow shallow trees in sequence. Tree 1 guesses 30 cartons; store 42 actually sells 42, an error of 12. Tree 2 is trained on that error and predicts +10; tree 3 is trained on the 2 that remain and predicts +2. The answer is the sum: 30 + 10 + 2 = 42.

Here is the same relay on house prices, the example the rest of this section uses.

With toy numbers, for a house that really sold for £580,000:

Tree 1 makes a rough guess: £500,000. It is £80,000 too low.
Tree 2 is not trained on prices at all. It is trained on what tree 1 got wrong, and predicts +£70,000. Now the total is £570,000, £10,000 short.
Tree 3 is trained on the £10,000 that is left, and predicts +£10,000.
The prediction is the sum: £500,000 + £70,000 + £10,000 = £580,000.

That is the relay, and Google’s Decision Forests course describes it the same way: each new weak model is trained to predict the errors of the current strong model and then added to it. Real libraries add one refinement my toy skips: each tree’s contribution is scaled down by a learning rate (a “shrinkage” factor; BigQuery ML’s LEARN_RATE defaults to 0.3), so they take many more, smaller steps, which overfits less.

one decision tree versus gradient-boosted trees for a house price: the single tree asks yes or no questions and lands on a leaf such as GBP 500k; boosted trees work as a relay, where tree 1 guesses GBP 500k, tree 2 predicts the GBP 80k left over and adds GBP 70k, tree 3 adds the last GBP 10k, and the sum is GBP 580k

The two libraries you will hear about most:

XGBoost (“extreme gradient boosting”), described by Tianqi Chen and Carlos Guestrin in XGBoost: A Scalable Tree Boosting System (2016). BigQuery ML’s boosted tree models are trained using the XGBoost library. XGBoost is still an ensemble of decision trees; what the paper adds is a regularised objective (a penalty on complex trees) and a sparsity-aware split finder that learns a default direction at each split, so a row with a missing value still finds its way down the tree.
LightGBM, which began as a Microsoft open-source project, introduced in the NIPS 2017 paper LightGBM: A Highly Efficient Gradient Boosting Decision Tree. Its README notes that the project moved from Microsoft/LightGBM to lightgbm-org/LightGBM in March 2026.

The misconception here is “deep learning beats everything, so use a neural network or an LLM.” On tables of business data, the measured evidence says otherwise. Grinsztajn, Oyallon and Varoquaux benchmarked deep learning against tree-based models on 45 tabular datasets in Why do tree-based models still outperform deep learning on tabular data? (2022) and found that tree-based models remained state of the art on medium-sized data (around 10,000 rows), even before counting their speed. That is one careful benchmark, not a law, but it matches what practitioners see.

My view: for a table of business rows, the sensible order is a logistic regression baseline first (for yes or no; linear regression for a number), because it trains in seconds, each feature has one readable weight, and it tells you whether your features carry any signal at all. Then gradient-boosted trees, which usually win on accuracy. Neural networks and LLMs come in for unstructured inputs: free text, images, audio, PDFs.

How does this apply to Acme? Sam’s service-risk model would start as a LOGISTIC_REG model, then move to a BOOSTED_TREE_CLASSIFIER if it scores better on the validation weeks. The first tree in the relay might learn something like this:

campaign_starts_next_week = 1?
  yes -> orders_growth_last_week > 1.05?
           yes -> otif_last_week < 96?   yes: high risk    no: medium risk
           no  -> low risk
  no  -> weeks_below_95_last_8 >= 2?     yes: medium risk  no: low risk

Later trees nudge those scores up or down for the cases the first tree got wrong. BigQuery ML also offers RANDOM_FOREST_CLASSIFIER, also trained with the XGBoost library, so at Acme, this is how it would work: supply trains both on the same weeks, compares them on the validation weeks, and keeps the winner.

How do you stop a model memorising the training rows?

The bottom line: regularisation charges the model for complexity, and cross-validation stops one lucky split from fooling you.

An overfitted model is brilliant on the rows it has seen and poor on new ones. Two tools fight it, for trees and for regression alike.

Regularisation adds a complexity tax to training. Picture a house-price formula with one weight per feature: price = w1 x area + w2 x bedrooms + ... + b. To fit every training house exactly, a model might set w1 = 999,999 and w2 = -999,998, two huge weights that cancel out on the training rows and go wild on anything new. Regularisation adds a penalty for large weights to the error the model is minimising, so huge weights stop paying off. There are two common forms:

L2 regularisation penalises the square of each weight, which shrinks all weights towards zero (Google’s crash course has a clear page on it).
L1 regularisation penalises the size of each weight, which tends to push useless ones, such as a door_colour column, all the way to exactly zero.

Trees have their own versions: cap the depth (BigQuery ML’s boosted trees default to a MAX_TREE_DEPTH of 6) and apply L1 and L2 penalties to the leaf values (its L2_REG defaults to 1.0).

Cross-validation deals with small data. Say you have only 500 houses. Hold back 15% (75 houses) for validation and you might, by bad luck, pick 75 mansions. k-fold cross-validation cuts the data into k equal slices and takes turns:

5-fold cross-validation on 500 houses: five slices A to E of 100 houses each

Run 1: train on A B C D   validate on E
Run 2: train on A B C E   validate on D
Run 3: train on A B D E   validate on C
Run 4: train on A C D E   validate on B
Run 5: train on B C D E   validate on A
Score = the average of the five validation scores; every house is validated exactly once.

There is a catch for dated rows: plain k-fold shuffles across time and leaks again. The time-aware version always trains on earlier slices and validates on the next one, so each training set grows forward in time; scikit-learn calls it TimeSeriesSplit. At Acme, this is how it would work: train on weeks 1 to 20 and validate on 21 to 24, then train on 1 to 24 and validate on 25 to 28, and so on, averaging the scores.

How do you forecast a number over time?

The bottom line: forecasting is regression where the rows are in time order and the past is part of the input. A forecasting model may only learn from the past, and it must be judged on weeks that come after everything it trained on.

Take a seaside shop’s monthly ice-cream sales (toy numbers). Two patterns jump out. Every summer sales peak and every winter they fall: that repeating cycle is seasonality. And each year’s July is a little higher than the last, 1,200 tubs in July 2025 and 1,260 in July 2026, because the shop is growing: that slow drift is the trend. What is left after trend and seasonality is noise. A forecast for next July starts from those two patterns, and the number of steps ahead you predict is the horizon: one month, three months, a year.

The misconception is that a forecast is just a smarter average. It is regression, with one extra rule: the rows have an order, and the order carries information. That is why the time-ordered split earlier in this post is not optional for forecasting; it is the whole game.

Univariate or multivariate: what may the model look at?

A univariate forecast uses one column: the series’ own past. The shop’s model sees only past monthly sales and extends the trend and the seasonality. Classic statistical methods such as ARIMA work this way.

A multivariate forecast adds drivers (also called covariates or external regressors): price, promotions, holidays, temperature. If the shop runs a two-for-one offer in May, a model that sees a promo_planned column can learn what offers do; a univariate model just sees a strange May.

Where people get tripped up is the catch: to use a driver in a forecast, you must know its future values when you make the forecast. The shop’s promotion calendar for next summer exists, because the shop sets it, so it is a usable driver. Next August’s temperature does not exist yet; to use it you would first have to forecast the weather months ahead, and that forecast is worse than useless. This is the leakage test from earlier applied to the future. BigQuery’s AI.FORECAST function makes the distinction explicit: in multivariate mode it has separate arguments for past covariates, known only historically, and future covariates, known for the forecast period too.

univariate versus multivariate forecasting and backtesting: a univariate model sees only past weekly orders; a multivariate model also sees drivers known in advance, such as the planned campaign weeks, but not drivers nobody knows yet, such as the weather months ahead; backtesting rolls the forecast origin forward, training on the past and scoring the next weeks each time, then averages the errors

How do lag features turn a series into a table?

Gradient-boosted trees know nothing about time. You can still use them for forecasting by turning the series into an ordinary table, one row per period, with lag features: earlier values of the series as columns.

month     sales  sales_lag_1  sales_lag_12  promo_planned   label: sales
Jun 2026  1,050        880          1,000              0           1,050
Jul 2026  1,260      1,050          1,200              0           1,260
Aug 2026     ?       1,260          1,230              1               ?

sales_lag_1 is last month’s sales and sales_lag_12 is the same month last year, which carries the seasonality. Add rolling averages (the mean of the last three months) and the drivers you know in advance, and a boosted tree model learns the rest. One rule keeps this honest: a lag must be available at forecast time. To forecast three months ahead, sales_lag_1 for that month does not exist yet, so either use lags of three or more, or feed each forecast back in as the next lag and accept that errors compound.

Why must forecasts at different levels add up?

Retailers forecast at several levels at once: each store, each region, the whole country. Forecast each level separately and they rarely agree. With toy numbers, 500 store forecasts might add up to 120,000 units while the national model says 105,000. Which one does the warehouse plan for? Hierarchical forecasting produces forecasts that are coherent: stores add up to regions and regions add up to the national total. The process of making them agree is reconciliation. The simplest method, bottom-up, forecasts the lowest level and sums upwards; more advanced methods, such as MinT, combine the forecasts from every level, as Forecasting: Principles and Practice explains. In BigQuery ML, an ARIMA_PLUS model takes a HIERARCHICAL_TIME_SERIES_COLS option that aggregates and rolls up the forecasts along the columns you list, left-most column as the parent.

How do you test a forecast? Backtesting

You test a forecast by pretending it is an earlier date. Rewind to the end of June, train on everything up to then, forecast July, and compare with what really happened. Then roll forward a month: train up to July, forecast August. Repeat across several periods and average the errors. This is backtesting, also called evaluation on a rolling forecasting origin, because the point you forecast from rolls forward; it is the same idea as scikit-learn’s TimeSeriesSplit in the cross-validation section above. Several windows protect you from a model that got lucky in one quiet month.

Which numbers tell you a forecast is good?

The bottom line: MAE says how far off you are in units, RMSE punishes big misses, MAPE says how far off you are in percent, and bias says whether you are always too high or always too low.

Here are four weeks of a toy forecast. I define the error as forecast minus actual, so a negative error means the forecast was too low:

week   actual  forecast   error   |error|   % error
1         100        90     -10        10     10.0%
2         120       110     -10        10      8.3%
3          80        90     +10        10     12.5%
4         100        95      -5         5      5.0%

MAE  (mean absolute error)       = (10 + 10 + 10 + 5) / 4              = 8.75 units
RMSE (root mean squared error)   = sqrt((100 + 100 + 100 + 25) / 4)     = sqrt(81.25) = 9.01 units
MAPE (mean absolute % error)     = (10.0 + 8.3 + 12.5 + 5.0) / 4        = 9.0%
Bias (mean error)                = (-10 - 10 + 10 - 5) / 4              = -3.75 units, too low

Each one answers a different question:

MAE is the plain average miss, in the units the planner cares about.
RMSE squares the errors before averaging, so one big miss dominates. Compare a second model with errors of 0, 0, 0 and -35: its MAE is also 8.75, but its RMSE is sqrt(1,225 / 4) = 17.5, twice the first model’s. Use RMSE when a single large miss, such as an empty shelf in launch week, is what really costs you.
MAPE is unit-free, so it compares a big brand with a small one. Its weakness is low volumes: a region that sells 2 units against a forecast of 6 scores a 200% error, and a week with zero sales makes it infinite or undefined, a weakness Forecasting: Principles and Practice spells out.
Bias catches a model that is always on the same side. A 9% MAPE that swings above and below the actuals can be fine; a model that is 9% too low every single week empties the shelves all year.

What are time-series foundation models?

Just as LLMs were pretrained on huge amounts of text, researchers have pretrained models on huge collections of time series, so they can forecast a series they have never seen without any training on it. This is called zero-shot forecasting. Three you will meet:

TimesFM from Google Research, described in A decoder-only foundation model for time-series forecasting (ICML 2024). The Google Research blog says it was pretrained on 100 billion real-world time points, and the paper reports zero-shot accuracy that comes close to supervised models trained on each dataset.
Chronos, from AWS AI Labs: Chronos: Learning the Language of Time Series (2024) turns values into tokens from a fixed vocabulary and trains language-model architectures on them.
Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting (2023), an open model built around lag features.

Before them came deep learning forecasters trained per dataset, such as DeepAR (2017) and Temporal Fusion Transformers (2019, written as part of an internship with Google Cloud AI Research).

On Google Cloud you can use TimesFM with no model to train: BigQuery’s AI.FORECAST runs the built-in TimesFM model. As of November 2026 the default is TimesFM 2.5; TimesFM 3.0 and multivariate forecasting with covariates are in Preview.

When do they win, and when does a tuned model win? My view, hedged: a foundation model is a strong first answer when you have little history (a new product, the cold start problem) or thousands of series and no time to build a pipeline for each. A model trained on your own data, such as ARIMA_PLUS_XREG or boosted trees on lag features, has a better chance when you have long history and strong drivers that only you know, such as your own campaign calendar. Neither is a rule; run both through the same backtest and let the errors decide.

How does this apply to Acme?

At Acme, this is how it would work. Sam needs a weekly forecast of units_ordered from supply_service for each brand (Acme Pure, Verde and Harvest Row) and region, eight weeks ahead. The obvious driver is marketing’s campaign plan. Supply already sees when campaigns run through the campaign_calendar view over campaign_performance (Prerequisite 2), and campaigns are planned weeks in advance, so campaign_active passes the known-future test. It matters: in the Refill Launch weeks Acme Pure’s orders in Western Europe rose by about 22% (Prerequisite 2), which a univariate model trained on the quiet weeks before could not have seen coming.

BigQuery ML’s ARIMA_PLUS model handles one series or many (TIME_SERIES_ID_COL), models trend with ARIMA, finds seasonal patterns, and can add holiday effects (HOLIDAY_REGION). Its multivariate sibling, ARIMA_PLUS_XREG, adds drivers such as campaign_active:

CREATE OR REPLACE MODEL `acme_supply_ml.orders_forecast_v1`
OPTIONS (
  model_type = 'ARIMA_PLUS_XREG',
  time_series_timestamp_col = 'week_start',
  time_series_data_col = 'units_ordered',
  time_series_id_col = ['brand', 'region'],   -- one series per brand and region
  data_frequency = 'WEEKLY'
) AS
SELECT week_start, brand, region, units_ordered, campaign_active
FROM `acme_supply_ml.weekly_orders`;          -- supply_service joined to campaign_calendar

SELECT brand, region, forecast_timestamp, forecast_value,
       prediction_interval_lower_bound, prediction_interval_upper_bound
FROM ML.FORECAST(MODEL `acme_supply_ml.orders_forecast_v1`,
  STRUCT(8 AS horizon, 0.9 AS confidence_level),
  (SELECT * FROM `acme_supply_ml.planned_weeks`));   -- the next 8 weeks with campaign_active

ML.FORECAST returns a forecast value and a prediction interval for each week; for an ARIMA_PLUS_XREG model you pass a table holding the drivers for the forecast weeks, which is the known-future rule in SQL. ML.EXPLAIN_FORECAST splits an ARIMA_PLUS forecast into its trend, seasonal and holiday components, which is what Sam would show a planner who asks why. Before trusting it, Sam would backtest it on rolling windows of past weeks and watch MAE and bias, by brand.

The agent pattern is the same as for the risk score. The forecasting model computes; Gemini calls it as a tool, compares scenarios and explains. The supply agent gets a narrow tool, run_forecast(brand, weeks, campaign_plan, region="Western Europe"): the model supplies a brand, a number of weeks and the planned campaign weeks; code builds the future campaign_active rows, runs ML.FORECAST with query parameters, and returns numbers. Never paste thousands of rows of history into a prompt and ask an LLM to continue the series. A what-if turn, with toy forecast numbers, would go like this:

1. Maria: "If we run the Refill Launch again for four weeks from 7 December, how many extra
   orders should supply expect for Acme Pure in Western Europe, and can it cope?"
2. Router -> supply agent (a supply measure: delegate).
3. run_forecast("Acme Pure", 8, campaign_plan=["2026-12-07", "2026-12-14", "2026-12-21",
   "2026-12-28"]) -> campaign weeks average 16,950 units (90% interval 15,600 to 18,300)
4. run_forecast("Acme Pure", 8, campaign_plan=[]) -> the same weeks average 13,800 units
5. score_service_risk("Acme Pure", "Western Europe") -> {"p_miss_95": 0.81, "zone": "act"}
6. Gemini: "Supply's forecast puts orders at about 16,950 a week during the campaign against
   13,800 without it, roughly 12,600 extra units over the four weeks. The service-risk model
   gives an 81% chance of missing the 95% OTIF target, so I have prepared a replan request for
   Sam to approve."

Every number in the answer came from a tool. Gemini chose the two scenarios, subtracted (16,950 - 13,800) x 4 = 12,600, which code can check, and wrote the sentence. The replan still pauses for Sam, as in Part 9.

One honest gap: Acme has run only a handful of big launches, so the model has few examples of what a campaign does to orders. Its prediction intervals will be wide in campaign weeks, and the agent should quote the interval, not just the middle number.

Why is 98% accuracy a trap?

The bottom line: when the thing you care about is rare, a model that always says “no” scores high accuracy and is useless. Measure the rare class directly.

A hospital screens 1,000 patients for an early-stage disease. In reality 20 are sick (2%) and 980 are healthy. A lazy model says “healthy” to everyone. It is right 980 times out of 1,000: 98% accuracy, and every sick patient goes home untreated. Accuracy (the share of all predictions that were right) is misleading whenever one class is rare, and the crash course warns against it for imbalanced data for exactly this reason.

How does this apply to Acme? Suppose (toy numbers) supply has 1,000 brand, region and week rows of history and 60 of them missed 95% OTIF: 6%. A model that always answers “the target will be met” is 94% accurate and has never warned Sam about anything.

You can also help the model during training, because with so few positives it barely notices them. Two fixes, both in the crash course’s page on imbalanced data:

Class weights: make each rare row count more in the training error. BigQuery ML’s AUTO_CLASS_WEIGHTS = TRUE uses the formula total_rows / (rows_in_class x number_of_classes). For the hospital: sick rows get 1,000 / (20 x 2) = 25, healthy rows get 1,000 / (980 x 2) = 0.51, so one sick patient counts as much as 49 healthy ones (980 / 20). For Acme’s toy history: missed weeks get 1,000 / (60 x 2) = 8.33 and good weeks 1,000 / (940 x 2) = 0.53.
Resampling: show the model fewer majority rows (downsampling) or repeat the minority rows (oversampling).

What do precision, recall and F1 measure?

The bottom line: precision asks “when the model says yes, how often is it right?”; recall asks “of all the real cases, how many did it find?”; F1 is one number that is only high when both are.

Back to the hospital. A real model scans the 1,000 patients and flags 30 as sick. Biopsies show 15 of the 30 are truly sick and 15 were false alarms. There were 20 sick patients in total, so it caught 15 and missed 5.

Precision looks only at the 30 the model flagged: 15 / 30 = 50%. Memory aid: precision is how precise the alarms are; no false alarms means 100% precision.
Recall looks only at the 20 truly sick: 15 / 20 = 75%. Memory aid: recall is how many real cases you recovered; no misses means 100% recall.

The four counts behind these numbers form a confusion matrix: true positives (flagged and sick, 15), false positives (flagged but healthy, 15), false negatives (missed, 5) and true negatives (correctly left alone, 965). Its accuracy is (15 + 965) / 1,000 = 98%, exactly the same as the lazy model that caught nobody.

precision versus recall for a screening test on 1,000 patients with 20 truly sick: of the 30 the model flags, 15 are sick, so precision is 15 / 30 = 50%; of the 20 sick, it catches 15 and misses 5, so recall is 15 / 20 = 75%; F1 is 0.60, and accuracy is a flattering 98%

Why does F1 use the harmonic mean?

You often want one number to compare two models. Take a lazy Model A that flags all 1,000 patients. Its recall is 100% (it caught all 20), and its precision is 20 / 1,000 = 2%. A normal average gives (100% + 2%) / 2 = 51%, a respectable score for a useless model.

The F1 score is the harmonic mean of precision and recall, a kind of average that is pulled towards the smaller number:

F1 = 2 x precision x recall / (precision + recall)

Lazy Model A:      2 x 0.02 x 1.00 / (0.02 + 1.00) = 0.04 / 1.02 = 0.039   -> 3.9%
Screening model:   2 x 0.50 x 0.75 / (0.50 + 0.75) = 0.75 / 1.25 = 0.60    -> 60%

To score a high F1, precision and recall must both be high; if either falls towards zero, F1 falls with it.

How does this apply to Acme? With toy numbers again: the test set is the newest 200 brand, region and week rows, and 12 of them missed 95%. At the default cutoff of 0.5, Sam’s model flags 10 rows, and 8 of those truly missed:

precision = 8 / 10 = 80%       (when it warns Sam, it is right 4 times in 5)
recall    = 8 / 12 = 67%       (it caught 8 of the 12 real misses; 4 slipped through)
F1        = 2 x 0.80 x 0.67 / (0.80 + 0.67) = 0.73
accuracy  = (8 + 186) / 200 = 97%, against 94% for "always fine": accuracy barely moves

What does ROC-AUC measure?

The bottom line: ROC-AUC grades how well the model ranks real cases above non-cases, across every possible cutoff at once: 0.5 is a coin flip, 1.0 is a perfect ranking.

Here is the secret behind every classifier, from logistic regression to boosted trees: the model never outputs “sick” or “healthy”. It outputs a score between 0 and 1, and something else turns that score into a decision. Precision, recall and F1 all depend on where you put that line, and they all change when you move it. So how do you judge the scores themselves?

ROC-AUC (the area under the receiver operating characteristic curve) answers that. The ROC curve plots, for every possible cutoff, the share of real cases caught (recall, also called the true positive rate) against the share of non-cases wrongly flagged (the false positive rate). The area under it has a plain meaning, which Google’s crash course states directly: it is the probability that the model, given one randomly chosen positive and one randomly chosen negative example, ranks the positive higher.

Work it out on five patients. The model gives them these scores: A 5% (healthy), B 18% (healthy), C 35% (sick), D 60% (healthy), E 90% (sick). There are 2 sick x 3 healthy = 6 sick-healthy pairs. Count the pairs where the sick patient has the higher score:

C 35% vs A 5%   yes      E 90% vs A 5%   yes
C 35% vs B 18%  yes      E 90% vs B 18%  yes
C 35% vs D 60%  no       E 90% vs D 60%  yes

AUC = 5 right pairs / 6 pairs = 0.83

One pair is out of order (healthy D outscored sick C), so the AUC is 0.83. If every sick patient outscored every healthy one, it would be 1.0. If the scores were shuffled at random, about half the pairs would come out right: 0.5, a coin flip. In that case no cutoff can rescue the model; you need better features, not a better threshold.

How does this apply to Acme? On the 200 test rows there are 12 missed weeks and 188 good ones, so 12 x 188 = 2,256 pairs. A toy AUC of 0.88 would mean that in about 1,985 of those pairs, the missed week got the higher risk score. BigQuery ML reports this as roc_auc in ML.EVALUATE, and ML.ROC_CURVE returns the recall and false positive rate at a range of thresholds so you can draw the curve.

Where should the cutoff go?

The bottom line: the threshold is a business decision, not a statistical one. Put it where the cost of a false alarm and the cost of a miss balance for your use.

The model hands you a score; a line of code turns it into an action: if score >= THRESHOLD: flag(). Most tools default to 0.5 (BigQuery ML’s ML.PREDICT and ML.EVALUATE both do), and the crash course is blunt that 0.5 is a poor choice when one kind of mistake costs more than the other or the classes are imbalanced. Move the line on the five patients and watch the trade-off:

five patients ranked by risk score with three possible cutoffs: at 20% the model flags C, D and E and catches both sick patients; at 50% it flags D and E and misses C; at 80% it flags only E with no false alarm; ROC-AUC ignores the cutoff and counts how often a sick patient outranks a healthy one: 5 of 6 pairs, 0.83

Moving the cutoff left (lower) catches more real cases, so recall goes up, but it sweeps in more false alarms, so precision goes down. Moving it right does the opposite. The business cost decides:

If a false alarm costs a second blood test and a miss costs a patient’s health, set the cutoff low. That is a high-recall system: initial medical screening, or luggage screening where a person checks every alarm.
If a false alarm means automatically freezing a customer’s bank card at the till, set it high. That is a high-precision system. A spam filter is the classic example; the crash course notes that misfiling a legitimate email can be worse than letting a spam email into the inbox. Your job offer landing in spam is a disaster; one stray promotion in your inbox costs a second to delete.

Why use two thresholds and a grey zone?

You do not have to pick one line. Many systems use two thresholds to make three zones. A card issuer scoring transactions for fraud might work like this (the numbers are illustrative):

score below 0.30        -> approve the payment
score 0.30 to 0.90      -> grey zone: approve, but text the customer "Was this you?"
                           or send it to a fraud analyst
score 0.90 and above    -> block the card at once

The top zone is high precision, because blocking a good customer at the till is expensive. The grey zone keeps recall high without punishing anyone: a cheap check, or a person, handles the uncertain middle.

What is a neural network, and how does it lead to an LLM?

The bottom line: a neural network stacks many small weighted sums in layers, with a simple non-linear switch after each one, so it can build its own features from raw inputs. That is why it wins on pixels and words, where nobody can hand-engineer the columns, and why it is usually overkill for a clean table.

Linear regression is one weighted sum. A neural network chains many of them. Each neuron (or node) does what linear regression does, a weighted sum of its inputs plus a bias, and passes the result on. Neurons are arranged in layers:

The input layer holds the raw numbers: three columns from a spreadsheet, or the brightness of every pixel in an X-ray.
The hidden layers sit in the middle. They are called hidden because you only ever feed the inputs and read the output; the numbers in between are internal to the calculation.
The output layer gives the answer: £580,000 for a house, 0.85 for a patient, or a score for every possible next token in an LLM.

A network with one hidden layer is shallow. Google’s ML glossary defines a deep model, or deep neural network, as one with more than one hidden layer, which is all the “deep” in deep learning means.

a small neural network for a house price: three inputs, two hidden layers of four neurons and one output; every neuron takes a weighted sum of the layer before plus a bias and passes it through an activation function such as ReLU, which turns negatives into 0; without that step the stacked layers collapse into one straight-line formula; more than one hidden layer makes the network deep

Why does every neuron need an activation function?

Where people get tripped up is assuming that more layers automatically mean more power. Without one extra step, they do not. Take a one-input network with no extras: layer 1 multiplies by 3, layer 2 multiplies by 2. Feed in x and you get 2 x (3x) = 6x, which a single layer could have done. Stack a hundred such layers and you still have one straight line. Google’s crash course says it plainly: linear operations performed on linear operations are still linear.

The fix is an activation function, a small non-linear step after each neuron’s sum. The most popular is ReLU (rectified linear unit): if the input is negative or zero the output is 0, otherwise the output equals the input. So ReLU(-3) = 0 and ReLU(3) = 3. That bend at zero is enough: with ReLU between layers, a network can learn curves, thresholds and interactions that no single weighted sum can. (The sigmoid from logistic regression is another activation function; large language models use variants too, such as the GeGLU used by Gemma 2.)

What do the layers learn?

Each layer builds on the one before, which is where the idea of a hierarchy comes from. The classic evidence is Zeiler and Fergus’s Visualizing and Understanding Convolutional Networks (2013), which looked inside an image classifier: layer 2 responded to corners and edge and colour combinations, layer 3 to textures such as mesh patterns, layer 4 to more specific parts such as dog faces, and layer 5 to whole objects. An X-ray model would follow the same pattern: edges, then shapes, then ribs and lungs, then the shadow a radiologist cares about.

For LLMs, I would hold that picture more loosely. Interpretability research suggests the middle of a model is where many abstract concept features live; Anthropic’s 2024 feature study of Claude 3 Sonnet, Scaling Monosemanticity, deliberately looked at the middle layer, halfway through the model. But it is an active research area, not a settled map, and Prerequisite 4 explains what interpretability can and cannot tell you today.

How many layers is “deep” in practice? The layer counts of Gemini, Claude and GPT models are not public. Open models publish theirs: the Gemma 2 technical report lists 26 layers for the 2B model, 42 for 9B and 46 for 27B.

What are the four families of neural network?

Engineers have designed different wirings, called architectures, for different shapes of data. Four families cover almost everything you will meet:

MLP (multi-layer perceptron), also called a feed-forward or dense network. Every neuron connects to every neuron in the next layer, as in the diagram above. On its own it is the textbook network; on business tables it usually loses to gradient-boosted trees. It lives on inside LLMs, though: in the original Transformer, every layer pairs a self-attention step with a small fully connected feed-forward network.
CNN (convolutional neural network). Built for grids such as photos, scans and video frames. Instead of connecting every neuron to every pixel, it slides a small filter (for example 3 x 3 pixels) across the image, so it can spot the same edge or shape wherever it appears. The glossary notes CNNs’ success in image recognition.
RNN (recurrent neural network) and LSTM. Built for sequences. An RNN reads one step at a time and feeds its hidden state from each step into the next, like a running memory. Two weaknesses pushed it aside for language: it cannot process a long sequence in parallel, and it suffers from the vanishing gradient problem, which makes long-range dependencies hard to learn. The LSTM (long short-term memory) cell was designed to ease that, with gates that decide what to keep and what to forget.
Transformer. Introduced in Attention Is All You Need (Vaswani and others, 2017), which proposed an architecture “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely”. Its core is self-attention: every token in the input looks at every other token and pulls in what is relevant, all in parallel. In “The courier lost the parcel because it was damaged”, attention lets the representation of it draw on parcel. Because the whole sequence is processed at once, Transformers train efficiently on modern accelerators, and they are the architecture behind today’s LLMs and most embedding models. The Gemini technical report says Gemini models build on Transformer decoders.

So an LLM is not a different kind of thing from the models above. It is a very deep neural network, a stack of Transformer layers with billions of weights, trained on enormous amounts of text to predict the next token.

Which model wins which kind of data?

The bottom line: match the model to the shape of the data, and start with the cheapest model that fits that shape.

which model to start with for which data: tables start with linear or logistic regression and then gradient-boosted trees; time series start with lag features and boosted trees or a forecasting model such as ARIMA_PLUS; images start with a convolutional network or a vision transformer; text, code and documents go to a Transformer LLM with an embedding model; data with no labels goes to clustering or anomaly detection

How does this apply to Acme? Sam’s service-risk data is a table, so it starts at the top row: logistic regression, then boosted trees. BigQuery ML also offers a DNN_CLASSIFIER, a neural network for tables, and it is worth a try only if the boosted trees plateau. The delivery notes Sam’s team writes when a week goes wrong are text, so summarising them belongs to the fourth row: an LLM. One question can need both, which is exactly what an agent is for.

How does this apply to Acme?

The bottom line: at Acme, supply would train the service-risk model in BigQuery ML, and the supply agent would call it through a narrow tool; the model scores, code picks the zone, a person approves any write, and the LLM explains.

How would supply train and test the model in BigQuery ML?

At Acme, this is how it would work. Supply owns the OTIF numbers, so supply owns this model too. BigQuery can read supply_service through its Iceberg metadata (Prerequisite 2 explains the UniForm set-up), so supply could train the model in BigQuery ML, which lets you create and run models with SQL. Databricks has its own ML tools, and the platform is supply’s choice; I use BigQuery ML here because the whole flow is a handful of SQL statements.

Assume supply has built a feature table, acme_supply_ml.service_risk_weeks, with one row per brand, region and week, the features from earlier and the label otif_below_95. First train, keeping the newest weeks out entirely as the test set:

CREATE OR REPLACE MODEL `acme_supply_ml.service_risk_v1`
OPTIONS (
  model_type = 'BOOSTED_TREE_CLASSIFIER',   -- trained with XGBoost; try LOGISTIC_REG first
  input_label_cols = ['otif_below_95'],
  auto_class_weights = TRUE,                -- misses are rare: weight them up
  max_tree_depth = 4,                       -- a complexity cap (the default is 6)
  data_split_method = 'SEQ',                -- split by time, not at random
  data_split_col = 'week_ts',               -- excluded from the features automatically
  data_split_eval_fraction = 0.2            -- the newest 20% of these rows: validation
) AS
SELECT brand, region, otif_last_week, orders_growth_last_week,
       campaign_starts_next_week, weeks_below_95_last_8,
       TIMESTAMP(week_start) AS week_ts,     -- SEQ needs TIMESTAMP, not DATE
       otif_below_95
FROM `acme_supply_ml.service_risk_weeks`
WHERE week_start < '2026-07-06';            -- later weeks are the test set, never seen here

In BigQuery ML’s words, the held-back 20% is evaluation data, used to avoid overfitting through early stopping; in this post’s words it plays the validation role. Then score the test weeks once, at the threshold you are considering:

SELECT precision, recall, f1_score, roc_auc, accuracy
FROM ML.EVALUATE(MODEL `acme_supply_ml.service_risk_v1`,
  (SELECT * FROM `acme_supply_ml.service_risk_weeks` WHERE week_start >= '2026-07-06'),
  STRUCT(0.5 AS threshold));

SELECT threshold, recall, false_positive_rate, true_positives, false_positives
FROM ML.ROC_CURVE(MODEL `acme_supply_ml.service_risk_v1`,
  (SELECT * FROM `acme_supply_ml.service_risk_weeks` WHERE week_start >= '2026-07-06'));

ML.EVALUATE returns precision, recall, accuracy, f1_score, log_loss and roc_auc, and accepts a custom THRESHOLD for binary classifiers. Pass the test rows explicitly: calling it with no input data returns the metrics from training, and that form is deprecated. ML.ROC_CURVE returns one row per threshold, which is exactly what Sam needs to pick the two cutoffs. ML.PREDICT then scores new rows, returning predicted_otif_below_95 and predicted_otif_below_95_probs, the probability of each label.

How does the agent use the score?

The bottom line: the agent never estimates the probability. It calls a narrow tool that runs ML.PREDICT, code puts the score in a zone, and the model explains the result in words.

This is the narrow-tool pattern from Part 2: the model fills in a brand and a region; the SQL, the model name and the thresholds live in code. A sketch of the tool the supply agent would have:

from google.cloud import bigquery

LOW, HIGH = 0.30, 0.80   # chosen by Sam from ML.ROC_CURVE on the test weeks, not by the LLM

SQL = """
SELECT (SELECT p.prob FROM UNNEST(predicted_otif_below_95_probs) AS p WHERE p.label = 1) AS p_miss
FROM ML.PREDICT(MODEL `acme_supply_ml.service_risk_v1`,
  (SELECT * FROM `acme_supply_ml.service_risk_next_week` WHERE brand = @brand AND region = @region))
"""

def score_service_risk(brand: str, region: str) -> dict:
    """Probability that OTIF falls below 95% next week, from supply's model, with a zone."""
    params = [bigquery.ScalarQueryParameter("brand", "STRING", brand),
              bigquery.ScalarQueryParameter("region", "STRING", region)]
    rows = list(bigquery.Client().query(SQL, job_config=bigquery.QueryJobConfig(
        query_parameters=params)).result())
    if not rows:
        return {"status": "error", "message": f"no row for {brand} in {region} next week"}
    p = round(rows[0].p_miss, 2)
    zone = "act" if p >= HIGH else "review" if p >= LOW else "low"
    return {"status": "ok", "brand": brand, "region": region, "p_miss_95": p,
            "zone": zone, "model": "service_risk_v1"}

The brand and region travel as a parameterised query, never pasted into the SQL. And the zone is decided by two constants in code, the two thresholds from the previous section.

Here is how a turn would go, with toy scores:

1. Maria: "Refill Launch restarts next week in Western Europe and Southern Europe.
   Will supply keep up?"
2. Router (Part 5): a supply question about a supply measure -> delegate to the supply agent.
3. Supply agent calls score_service_risk("Acme Pure", "Western Europe")
   -> {"p_miss_95": 0.81, "zone": "act", ...}
4. Supply agent calls score_service_risk("Acme Pure", "Southern Europe")
   -> {"p_miss_95": 0.44, "zone": "review", ...}
5. Code, not the model, maps the zones to next steps:
   act    -> the agent may propose trigger_sop_replan; the run pauses for Sam's confirmation
   review -> the agent flags the region to Sam with the score and the features behind it
   low    -> nothing to do
6. Answer: "Supply's service-risk model puts the chance that Acme Pure misses the 95% OTIF
   target next week at 81% in Western Europe and 44% in Southern Europe. I have prepared a
   replan request for Western Europe for Sam to approve, and flagged Southern Europe to him
   for review."

Who decided what? The model chose to call the tool, called it twice, and wrote the sentence. The probability came from the predictive model. The zones came from two constants in code. The replan, the one action that writes data, still pauses for a person, exactly as in Part 9. The grey zone is a human-approval step by design.

This model is not part of the series’ build; it shows how a predictive model would plug into it. The thinking is not new to the series, though. Model Armor in Part 9 runs in audit mode first so you can count false positives before blocking anything, which is precision thinking. The judges in Part 10 are classifiers checked against human labels, and recall at k in Part 13.75 is the recall from this post applied to retrieval.

One honest gap: a probability is only useful if it is calibrated, and class weights, which help the model notice rare rows, also push its raw scores upwards. A 0.81 from a weighted model may not mean “81 in 100 such weeks miss”. That is why Sam picks the thresholds from ML.ROC_CURVE on the test weeks rather than reading the scores as literal odds, and why the answer above says “the model puts the chance at” rather than stating it as fact.

Does this need an LLM at all? A litmus test

The bottom line: ask the cheapest question first, and reach for an LLM only for the parts that are language. Run through these in order and stop at the first yes:

1. Can a rule in code answer it?               "Which weeks were below 95% OTIF?"
                                                -> SQL. No model at all.
2. Is it a known formula over known rows?      "What did the Refill Launch cost per click?"
                                                -> SQL or a metric view (Prerequisite 2).
3. Is it a score or a forecast for rows        "Which brand and region will miss 95% next week?"
   of a table, with past answers to learn from? -> a predictive model: logistic regression,
                                                   then gradient-boosted trees.
4. Are there no answers to learn from, only    "Which campaign weeks behave alike? Which is odd?"
   rows to group or check?                      -> clustering or anomaly detection.
5. Is the input free text, images or audio,    "Summarise why these 40 delivery notes were late."
   or is the job to explain, route or write?    -> a neural network: for text, an LLM.
Only then: an LLM that orchestrates the steps above through tools and explains the results.

Most real agent questions mix these. Maria’s question used step 3 for the scores, step 5 to write the answer, and code for the zones.

Lock it in

TERM               WHAT IT IS                                    SIMPLE EXAMPLE
Supervised         learn from rows that carry the answer         past sales with their prices
Unsupervised       find structure in rows with no answer         k-means: two shopper segments
Feature            an input column you choose and compute        sqft_per_bedroom = 1,000
Label              the answer column you want to predict         sold_price, otif_below_95
One-hot encoding   one 0 or 1 column per category value          is_detached = 1, is_flat = 0
Train / valid /    learn on, tune on, score once on              lessons, mock tests, real test
  test
Leakage            the model sees what it could not know         training on 22 Nov, testing on 21 Nov
Regularisation     a tax on complexity                           shrink w1 = 999,999 back down
k-fold CV          take turns validating on k slices             5 runs on 500 houses
Model families     linear, tree, distance and margin, neural,    regression, XGBoost, k-NN, CNN,
                   clustering                                    k-means
Decision tree      root, condition nodes, leaves                 credit score < 700? -> deny
Gini impurity      how mixed a group's labels are                5:5 = 0.50; pure = 0
Random forest      deep trees on bootstrap samples, vote         40, 44, 41 cartons -> 41.7
Gradient boosting  a relay of trees fixing the last error        500k + 70k + 10k = 580k
Accuracy trap      high accuracy from always saying "no"         98% on 20 sick in 1,000
Precision          right when it says yes                        15 / 30 = 50%
Recall             real cases found                              15 / 20 = 75%
F1                 harmonic mean of the two                      lazy model: 3.9%, not 51%
ROC-AUC            chance a positive outranks a negative         5 of 6 pairs = 0.83
Threshold          the score where action starts; a business     low for screening, high for
                   choice                                         blocking cards
Grey zone          the middle band between two thresholds,       "Was this you?" text; Sam reviews
                   sent to a cheap check or a person
Neural network     layers of weighted sums with activations      inputs, hidden layers, output
ReLU               0 for negatives, unchanged otherwise          ReLU(-3) = 0, ReLU(3) = 3
CNN                filters slid across a grid of pixels          edges, then shapes, then objects
RNN / LSTM         reads a sequence one step at a time           the pre-2017 approach to text
Transformer        self-attention over the whole sequence        "it" attends to "parcel"
Trend / season     slow drift / a repeating cycle                ice cream: growth / every July
Multivariate       series plus drivers known in advance          promo calendar yes, weather no
Lag feature        an earlier value of the series as a column    sales_lag_12 = last July
Reconciliation     make levels of a hierarchy add up             stores sum to the national total
Backtesting        forecast from a rolling origin, then score    train to June, test July, roll
MAE / RMSE         average miss / squares punish big misses      8.75 and 9.01; 8.75 and 17.5
MAPE / bias        % miss (breaks near zero) / always one side   9.0%; -3.75 means too low
Foundation model   pretrained forecaster, used zero-shot         TimesFM via AI.FORECAST

What can go wrong

Random splits on dated rows. The test score looks excellent and production disappoints. Split by time; in BigQuery ML, do not leave DATA_SPLIT_METHOD on its random default.
A feature from the future. If a column would not exist at prediction time, it is leakage, no matter how much it improves the score.
Celebrating accuracy. On rare events report precision, recall, F1 and ROC-AUC, and compare against the “always no” baseline.
Leaving the threshold at 0.5. It is a default, not a decision. Pick it from ML.ROC_CURVE and the business cost, and write down why.
Tuning on the test set. The moment you change a setting because of a test score, it has become a validation set and you need fresh test rows.
Letting the LLM restate the number. The agent should quote the tool’s probability, not rephrase it; the number check from Prerequisite 3 applies to scores too.
A driver from the future. A forecast that uses next quarter’s weather or actual sales of a campaign week has leaked; only plans and calendars you really know in advance qualify.
Judging a forecast by MAPE alone. Low-volume series blow it up, and it hides bias. Report MAE and bias by brand as well.
Forgetting that the world moves. A model trained before a new retailer contract or a new warehouse learns patterns that may no longer hold. That is data drift: re-run ML.EVALUATE on recent weeks on a schedule and retrain when the scores fall.

What did we learn?

Generative AI writes and traditional, predictive ML scores: when the job is a number or a probability for rows of a table, a small model trained on your own rows beats asking an LLM, and the agent calls it through a tool and explains the result.
Supervised learning learns from labelled rows, unsupervised learning finds groups and oddities in unlabelled ones, and reinforcement learning learns from rewards.
In predictive ML a feature is an input column and the label is the answer column; one-hot encoding, time aggregates and ratios turn raw data into useful features, and any column you could not know at prediction time is leakage.
Train, validation and test sets are lessons, mock tests and the real test; dated rows must be split by time, and regularisation and cross-validation keep a model from memorising.
Models come in families (linear, tree-based, distance and margin, neural networks, clustering); a decision tree picks each condition by how much it reduces impurity, a random forest averages many deep trees trained on bootstrap samples, and boosting chains shallow trees.
Gradient-boosted trees are a relay of small trees, each fixing the error left by the ones before it, and on medium-sized tables they remain the model family to beat, after a logistic regression baseline.
On rare events accuracy lies; precision, recall and F1 measure the rare class at one threshold, and ROC-AUC measures the ranking across all thresholds.
The threshold is a business decision; two thresholds make a grey zone that goes to a person, which at Acme becomes Sam’s approval step.
Forecasting is regression over ordered time: model trend and seasonality, use only drivers whose future values you know, turn the series into lag features for boosted trees, reconcile levels so they add up, backtest on rolling windows, and judge with MAE, RMSE, MAPE and bias; the forecasting model computes and Gemini calls it for what-if scenarios.
A neural network stacks weighted sums with activation functions such as ReLU, without which the layers collapse into one straight line; MLPs, CNNs, RNNs and Transformers suit different shapes of data, and an LLM is a very deep Transformer.
Match the model to the data: regression and boosted trees for tables, CNNs for images, Transformers for text, clustering when there are no labels.

What comes next?

Before the model internals, Prerequisite 2.75: MLOps and LLMOps shows how a model like this one stays right after launch: versioning, drift and retraining, and the same discipline for agents. After that, Prerequisite 3: How a model works opens up the Transformer from this post’s last family: how training sets its weights (the same predict, measure the loss and adjust loop that trains a logistic regression), what neurons and features are inside an LLM, what a token is, why the context window forgets, and which jobs this series never gives to an LLM. Keep this post’s split in mind as you read it: the LLM orchestrates and explains, and anything that is really a score belongs to a model like Sam’s.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and all of its data are fictional: the supply figures come from a generated demo dataset, and the house, loan, milk, shopper, ice-cream, hospital, card and Acme model and forecast figures are toy numbers chosen for teaching.