Databricks on Google Cloud, Part 9: Commercials, quotas and operations

The first eight parts of this series built a Databricks platform on Google Cloud: the account and workspaces, identity, networking, storage, governance, the platform services and the links to BigQuery and Gemini. None of that survives contact with production if three plain questions have no answer. Who pays, and for what? How big are we allowed to get before something refuses to start? And who looks after it on a Tuesday morning when a job fails?

This part answers those three questions. It is the least glamorous post in the series, and in my work with partners it is the one that most often decides whether a go-live date holds. A missing CPU quota stops a cluster as surely as a broken firewall rule, and the wrong person with the wrong billing role can delete every workspace in an account with one click.

The running example is the same as before. Acme Consumer Brands (fictional) runs its supply data in Databricks on Google Cloud in europe-west2, with classic compute in the projects acme-dbx-prod and acme-dbx-dev, the workspaces acme-prod-euw2-supply and acme-dev-euw2-shared, and the gold table acme_supply.supply_gold.supply_service. Here Acme gets its commercial set-up right, sizes its quotas for month-end, and puts a day-two routine in place.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: how Databricks on Google Cloud is bought, billed, limited and run, in three minutes. The narration is a synthetic voice; the script and animation are mine.

What are the key ideas?

Six terms carry this post. Each one gets its own section below.

Databricks unit (DBU). The unit Databricks bills in. The documentation defines it as “units of processing capability per hour based on VM instance type”. A bigger machine consumes more DBUs per hour.
SKU (stock-keeping unit). One line on a price list. Each kind of Databricks usage (jobs compute, all-purpose compute, SQL, serverless and so on) has its own SKU and its own price per DBU.
Marketplace order and entitlement. The purchase records Google Cloud Marketplace keeps when you subscribe to Databricks: an order on your Cloud Billing account, and an entitlement (the record that says you bought the product) on the vendor’s side.
Private offer. A Marketplace purchase with a custom price and terms for one buyer, used by companies with a negotiated contract.
Quota. A limit Google Cloud places on how much of a resource one project may use, such as CPUs per region. Classic Databricks clusters run in your project, so your quotas cap them.
Resource limit. A limit Databricks places on its own objects, such as 20 workspaces per account or 2,000 job tasks running at once per workspace.

Together they make four jobs: buy, meter, limit and run.

the four jobs this post covers for Acme: buy Databricks through a Google Cloud Marketplace order on the Premium or Enterprise plan; meter usage in DBUs that land on one Google Cloud bill and in the system.billing.usage table; respect three kinds of limit, Google Cloud quotas in the workspace project, Databricks resource limits and design-time ceilings; and run the platform day to day with logs, system tables, budgets and support

Part 1 explains the account, workspaces and projects these jobs act on, and Part 2 explains the identities.

Which plan do you need: Premium or Enterprise?

The plan does not change the engine; it changes which security and network controls you may switch on. So you choose it from your network and key requirements, before the first workspace exists.

Think of two trims of the same car. Both have the same engine, seats and steering wheel. The higher trim adds the alarm, the tracker and the locking fuel cap, and adding them after delivery is a trip back to the dealer, not a switch on the dashboard. Here the engine is notebooks, Spark and Unity Catalog, which are the same on both plans, and the extras are controls such as Private Service Connect and customer-managed keys, which only the Enterprise trim includes.

A Databricks plan (also called a pricing tier) decides which features you may switch on. As of October 2026, Databricks on Google Cloud has two: Premium and Enterprise. There used to be a third. The subscription documentation announced the end of life of Standard tier workspaces on Google Cloud: customers had until 1 October 2025 to upgrade, and any Standard workspace left on that date was upgraded to Premium automatically. If an older document or quote mentions Standard, that is why.

The obvious beginner question is “does the plan change the platform?” It does not change notebooks, Spark or Unity Catalog. It changes which security and network controls you may use, and on Google Cloud that matters for architecture, because two of the biggest network decisions are gated by plan:

Premium and Enterprise on Databricks on Google Cloud as of October 2026: customer-managed VPC, IP access lists and audit log delivery need Premium; Private Service Connect, customer-managed keys, serverless egress control and context-based ingress need Enterprise; the compliance security profile and enhanced security monitoring need Enterprise plus the Enhanced Security and Compliance add-on

Each row comes from the feature’s own page: customer-managed VPC, IP access lists and audit log delivery need Premium; Private Service Connect, customer-managed keys, serverless egress control and context-based ingress control need Enterprise, and so do the private endpoint rules that connect serverless compute to your own services. The IP access lists row deserves a note: Databricks now recommends context-based ingress “as your primary method for managing access” and migrating existing IP access lists to it, so a Premium workspace keeps the older front-end control while the recommended one needs Enterprise. The compliance security profile (a hardened image and extra controls needed to process data under standards such as HIPAA and PCI-DSS) and enhanced security monitoring need the Enterprise tier and the Enhanced Security and Compliance add-on, which is charged on top, as the requirements page says. One of those rows is not yet generally available: the page for configuring customer-managed keys carries a Public Preview label as of October 2026.

The practical consequence is the one Part 1 flagged: a customer-managed VPC needs Premium, and Private Service Connect needs Enterprise. Part 3 covers both, and Part 6 covers the keys. One small trap on upgrades: the same subscription page notes that access control settings are disabled by default on workspaces upgraded from Standard to Premium, and once an access control setting is enabled it cannot be disabled.

How does this apply to Acme? Acme’s security team wants Private Service Connect and customer-managed keys for production, the two extras that only the higher trim has, so Acme subscribes on Enterprise. My view: decide the plan from the network and key requirements, not from the price list, and decide it before the first workspace exists. Teams coming from Azure should check this early, because the plan a feature needs is not always the same on every cloud. The test I use, cheapest answer first, reads straight off the table above:

Which plan? Ask in this order:
1. Customer-managed VPC, IP access lists and audit log delivery are enough?   -> Premium
2. You need Private Service Connect, customer-managed keys, serverless
   egress control or context-based ingress?                                    -> Enterprise
3. You must process regulated data (HIPAA, PCI-DSS) with the compliance
   security profile?                        -> only then Enterprise plus the add-on

How does Databricks billing work?

You pay Google Cloud for everything; what changes between classic and serverless compute is whether the machines are a separate line or folded into the DBU price.

Picture two ways to move a sofa across town. You can hire a van from one firm and a driver by the hour from another, and pay both with the same card: two lines on the statement, and your van-hire loyalty discount applies only to the van. Or you can book a removal firm that brings its own van: one price that includes the vehicle, and your van-hire discount does not come into it. Classic compute is the van plus driver: the VMs (the van) are billed at your Compute Engine prices, and the DBUs (the driver’s hours) at the price of each SKU. Serverless is the removal firm: one DBU price that includes the machines. The card is your one Google Cloud bill. The usual mistake is to compare a serverless DBU price with a classic one and forget that the classic price leaves out the VMs.

Databricks bills in Databricks units (DBUs), defined in the key ideas above. The number of DBUs a workload consumes depends on the compute type (for example, all-purpose or jobs) and the Google Cloud machine type. Each kind of usage has its own SKU with its own price per DBU, and usage is metered per second. The Databricks on Google Cloud pricing page puts it as paying “for the compute resources you use at per second granularity”, with pay-as-you-go pricing or committed-use discounts.

Who do you pay? On Google Cloud, the answer is always Google Cloud. A Databricks account on Google Cloud is created by subscribing in Google Cloud Marketplace, and that subscription is tied to one Cloud Billing account (the Databricks documentation shows how to confirm which one through the Google order number). So Databricks DBUs appear on your Google Cloud bill, under the service name Databricks, next to your Compute Engine and Cloud Storage charges. Google Cloud collects the money and pays Databricks its share. You get one bill and one invoice, whichever kind of compute you use. Google’s Marketplace billing page adds that Marketplace charges appear as separate items in Cloud Billing reports and in the Cloud Billing export to BigQuery.

What changes with the kind of compute is which lines appear on that bill.

On classic compute, there are two kinds of line: the DBUs for Databricks’ software, and the Google Cloud infrastructure in your project (the VMs, disks and networking that the cluster uses), billed at your normal Google Cloud prices. Databricks’ own page on networking costs says that classic customers “manage and pay networking costs directly to GCP”. Strictly, these lines go to whichever billing account each workspace project is linked to, which is usually, but not necessarily, the one behind the Marketplace order; Part 1 draws that split.
On serverless compute, the VMs run in Databricks’ account, so their cost is folded into the serverless DBU price and there is no separate VM line. The serverless workspace documentation points out the consequence: “your cloud-provider credits or discounts do not apply to this usage”, meaning discounts that apply to Compute Engine in your own project (such as committed use discounts) do not reduce the cost of VMs that Databricks runs. You still pay it through the same Google Cloud bill, and the same page says a private offer with Databricks does apply to it.
In both cases, data in your own Cloud Storage buckets (storage, operations and network egress) is billed by Google Cloud as ordinary Cloud Storage usage.

How does this apply to Acme? Acme has both shapes. The nightly supply job on classic jobs compute is the van and driver: a DBU line, plus a Compute Engine line in acme-dbx-prod. The analysts’ serverless SQL is the removal firm: one DBU line, with the machines inside it. The next section costs the first of those line by line.

how Databricks on Google Cloud is billed: everything is paid to Google Cloud on one bill, and Google Cloud pays Databricks for the DBUs; on classic compute the bill has DBU lines plus the VMs, disks and networking in your project; on serverless compute the compute runs in Databricks' account and is inside the DBU price; your own Cloud Storage is always billed by Google Cloud

What does one job run cost, line by line?

The formula is short: a Databricks charge is DBUs consumed times the price per DBU of that SKU. The DBUs come from the machines. Start with a toy cluster and made-up rates, so the mechanism is visible before any real names appear:

Toy cluster, illustrative rates only (not real prices)
  2 nodes x 1.5 DBUs per node-hour x 4 hours  = 12 DBUs
  12 DBUs x US$0.20 per DBU                   = US$2.40   Databricks line
  2 VMs x US$0.50 per VM-hour x 4 hours       = US$4.00   Google Cloud line
  total                                       = US$6.40   on two lines of one bill

Three multiplications: nodes times rate times hours gives DBUs, DBUs times the SKU price gives the Databricks line, and VMs times the VM price times hours gives the Google Cloud line.

How does this apply to Acme? The same three multiplications, with a real shape. Take Acme’s nightly supply job, the job cluster from Part 1: one driver and 12 workers on n2-highmem-8, running for two and a half hours on jobs compute (the compute type for scheduled, non-interactive work).

one run of Acme's nightly supply job on classic jobs compute, costed on both lines of the bill: the Databricks line is the DBUs the 13 nodes consume over 2.5 hours, at the DBU rate of the machine type, times the price per DBU of the jobs SKU; the Google Cloud line is 13 VMs and their disks for 2.5 hours at Compute Engine prices; on serverless the same job has only a DBU line

I have kept the rates as symbols on purpose. The DBU rate per machine type is in Databricks’ pricing calculator, the list price per DBU is in the system.billing.list_prices table, and your own price depends on your contract. What the drawing shows is the shape: on classic compute, two lines that you optimise separately; on serverless, one line that includes the machines. One more detail from Part 1 belongs here: idle VMs in an instance pool (a set of ready VMs that clusters draw from) cost no DBUs, but Google still bills the VMs.

What do the billing system tables look like?

Inside Databricks, the system table system.billing.usage records every DBU by workspace, SKU and tag, and system.billing.list_prices holds the list prices, so you can build your own cost reports. A system table is a read-only table that Databricks maintains for you in the system catalog of Unity Catalog. The billable usage reference says the data is routed to all regions, so you see the whole account’s usage from any workspace, and records typically arrive within 12 hours. Here is what a few rows look like:

a snapshot of system.billing.usage for Acme with illustrative values: one row per usage record, with the date, workspace ID, billing origin product such as JOBS, SQL, INTERACTIVE, PREDICTIVE_OPTIMIZATION and ALL_PURPOSE, whether the workload was serverless, the DBUs used and the custom tags; usage_unit is DBU and is_serverless sits inside the product_features column

The columns that matter most are sku_name (the SKU), usage_quantity (the amount, in the unit named by usage_unit, usually DBU), billing_origin_product (which product caused the usage), usage_metadata (the cluster, job, warehouse or endpoint behind it), identity_metadata (who ran it) and custom_tags. Two columns on Google Cloud are always empty: metastore_id and the region fields in usage_metadata are documented as not populated on Databricks on Google Cloud.

To turn DBUs into money, join the two tables on the SKU and the time the price was valid. This is the documentation’s own example, filtered to Acme’s team tag (I have not run it against a live account):

SELECT SUM(u.usage_quantity * p.pricing.effective_list.default) AS list_cost_usd
FROM system.billing.usage AS u
JOIN system.billing.list_prices AS p
  ON p.sku_name = u.sku_name
 AND u.usage_end_time >= p.price_start_time
 AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)
WHERE u.custom_tags['team'] = 'supply'
  AND u.usage_date BETWEEN '2026-10-01' AND '2026-10-31';

One honest gap: list_prices holds list prices. The pricing table reference describes effective_list as the list price after temporary promotions, not your negotiated rate. So this query tells you where the DBUs go and roughly what they cost; the Google Cloud invoice tells you what you actually paid.

What does subscribing on Marketplace create?

Subscribing is a purchase with a paper trail on both sides: Google Cloud records an order and an entitlement, and Databricks creates the account they point to.

It works like subscribing to a magazine through your phone’s app store. The store charges your card and keeps the order in your purchase history; it tells the publisher “this person has paid”; you then create an account with the publisher, which checks the store’s record before it lets you read. The card is the Cloud Billing account, the purchase history is the Your Orders page, the “has paid” record is the entitlement, and the publisher account is the Databricks account.

You subscribe to Databricks through the Databricks listing on Google Cloud Marketplace, and during the subscription you choose the billing account it bills to. The person subscribing needs the Billing Administrator role (roles/billing.admin) on that billing account, and Viewer on a project linked to it, as the subscription guide lists. Companies with a contract use a Marketplace private offer instead of the self-serve trial. Either way, Acme then sees two kinds of charge: Google Cloud usage from the workspace project (virtual machines, disks, storage and networking) and Databricks usage from the subscription.

How does this apply to Acme? When Acme’s billing administrator subscribes, the same chain runs in five steps. Subscribing is a purchase, and Google Cloud Marketplace records it as a few distinct objects. It helps to know their names, because support teams and contracts refer to them.

The subscription click. On the Databricks listing, someone with the Billing Administrator role clicks Subscribe, picks the Cloud Billing account, accepts the terms and subscribes again. The Databricks steps then show a pop-up that says “Your order request has been sent to Databricks”.
An order. Marketplace records the purchase as an order against that billing account. Anyone with Billing Account Viewer can see it on the Your Orders page in Marketplace, which is also where plans are changed or cancelled. Each order has a Google order number, and the Databricks account console shows it, so you can confirm which billing account a Databricks subscription uses.
An account and an entitlement, on the provider’s side. Google’s partner documentation says that when a customer first buys from a vendor, “Cloud Marketplace creates an account resource that tracks the user’s relationship with you”, and when they choose a plan, “Google creates an entitlement, which indicates that the customer has bought your product”. Each has an ID created by Marketplace (the entitlement ID), sent to the vendor, which approves it through the Partner Procurement API. A private offer accepted later creates its own entitlement.
The Databricks account. Clicking Sign up with Databricks takes you to Databricks, where you choose the Google account to sign in with, an account name and a region. That creates the Databricks account, tied to the order, and Databricks approves the entitlement.
Usage. From then on Databricks reports DBU usage to Google, and it appears on the Cloud Billing account under the service name Databricks. Google’s billing page notes that a SaaS purchase can also add “an additional Google Cloud project, which typically has a project ID beginning with pr”, which Marketplace uses “to track and report your usage”. Leave it alone.

what happens when Acme subscribes to Databricks on Google Cloud Marketplace: a billing administrator clicks Subscribe and picks a Cloud Billing account; Marketplace records an order, visible on the Your Orders page, and creates an account and an entitlement request that it sends to Databricks; the administrator clicks Sign up with Databricks; Databricks creates the Databricks account, linked to that order, and approves the entitlement; workspaces are created in Acme's projects, and Databricks reports DBU usage so it appears on Acme's Google Cloud bill

The free trial gives credit for Databricks usage for 14 days, then becomes pay-as-you-go. The sign-up page is careful to say the credits apply only to Databricks usage: a classic workspace in your own project creates Google Cloud resources that Google bills you for during and after the trial.

How does a private offer work?

A private offer is a custom Marketplace price and set of terms that one vendor makes to one buyer. Think of a corporate hotel rate: the same hotel and the same booking site, but a price and terms negotiated for your company, unlocked only by the link your company was sent. Here the booking site is the Databricks listing on Marketplace, the negotiated rate is the private offer, and the link is the one Databricks sends. Databricks’ subscription pages tell contract customers not to use the self-serve steps and to ask their Databricks account team for a private offer instead. Google’s page on accepting a private offer fills in the mechanics:

The vendor sends a link. To accept, you need Billing Account Administrator on the billing account named in the offer, or both Billing Account User (roles/billing.user) and Consumer Procurement Order Administrator (roles/consumerprocurement.orderAdmin).
You review the price, the contract duration and the payment schedule (prepay, charged up front, or postpay, charged monthly), confirm the billing account, optionally turn on auto-renewal if the vendor offers it, and subscribe. The vendor then accepts, and the offer pricing starts on the contract start date.
You can accept a private offer only once. To change it, the vendor sends a new offer that replaces the active one.

How do you change the plan or the billing account?

Plan changes for a SaaS product live on Your Orders, under Change Plan, and Google’s manage billing page notes that some changes take effect immediately and some at the end of the subscription term.

Changing the billing account behind a subscription is a support request, not a setting. The Databricks subscription page says to “reach out to Google Cloud support”. Google describes the underlying process as an entitlement transfer: you need Billing Account Administrator, the vendor must agree and requests the transfer, every party approves by email, transfers “are not guaranteed and are subject to feasibility”, and the process “can take a week or more”. One warning on that page is easy to miss: during the transfer, do not buy a product from the same provider on the destination billing account, because that makes the transfer infeasible. Choose the billing account carefully the first time.

Who can cancel the subscription, and what happens if they do?

The order has one more property that matters more than any price: whoever can cancel it can delete every workspace in the account.

Think of a rented storage unit. Whoever holds the contract can end it at the front desk in a minute, and once it ends the facility clears the unit, whoever packed the boxes. The contract is the Marketplace order, the people allowed at the desk are those who hold the consumerprocurement.orders.cancel permission, and the boxes are every workspace in the Databricks account. The common assumption is that cancelling only stops the billing. It does not: it deletes.

who can cancel Acme's Databricks subscription and what follows: the Organisation Owner role on the acme organisation and the Billing Administrator role on the billing account both include consumerprocurement.orders.cancel; anyone holding it can open Your Orders and click Cancel purchase on the Databricks order, which cancels the subscription and deletes every workspace in the Databricks account, irreversibly; the fix is a short list of billing administrators, reviewed regularly

The warning in that diagram is the most important operational fact in this post. The Databricks documentation says the subscription can be cancelled by anyone with the consumerprocurement.orders.cancel permission on the billing account (Billing Administrators and Organisation Owners, by default), and that cancelling “deletes all workspaces in the Databricks account. Workspace deletion is not reversible.” The cancellation steps are short: open the listing, Manage orders, find the order, open its menu and click Cancel purchase. After that, nobody can reach the workspaces, notebooks or data; customer content in those workspaces is deleted within 30 days under the Databricks terms; and you cannot sign up again with the same email address. Contract customers cancel through their Databricks account team instead.

How does this apply to Acme? Acme keeps the list of people at the desk short: two named billing administrators on that billing account. Listing who holds roles on a billing account is one command (0X0X0X-0X0X0X-0X0X0X stands for Acme’s billing account ID; I have not run it):

gcloud billing accounts get-iam-policy 0X0X0X-0X0X0X-0X0X0X \
  --format="table(bindings.role, bindings.members)"

Remember to check the organisation as well, because Organisation Owner is inherited downwards (Prerequisite 2 explains the resource hierarchy).

Who supports what?

This is the question customers most often get wrong, because there are two companies and two support contracts. The Databricks support page for Google Cloud is explicit about where it starts: “The support process for Databricks on Google Cloud starts with Google support”, through Google Cloud Customer Care, and “Google Cloud and Databricks support work jointly on support issues”. A separate Databricks support contract, arranged with your Databricks account team, lets authorised contacts open cases with Databricks directly.

It works like a car bought from a dealer and still under the maker’s warranty. You take it to the dealer, and when the fault is deep in the engine the dealer brings in the maker; you do not have to work out who built which part before you walk in. The dealer is Google Cloud Customer Care, the maker is Databricks, and a Databricks support contract is like having the maker’s own hotline as well.

who supports what for Databricks on Google Cloud: the Databricks documentation says the support process starts with Google Cloud Customer Care and that Google Cloud and Databricks support work jointly; a Databricks support contract adds a direct route to Databricks; my rule of thumb sends Google Cloud resources such as quotas, VPC, Cloud NAT, Private Service Connect, IAM, organisation policies, Compute Engine capacity and the Marketplace order to Google, and Databricks software such as the workspace, Spark, Unity Catalog, SQL warehouses, jobs and serving to Databricks

How does this apply to Acme? Acme’s runbook still sorts a problem before it opens a ticket, so the right side picks it up quickly. My rule of thumb, which is not a documented split: problems with Google Cloud resources (quotas, capacity, VPC, Cloud NAT, Private Service Connect, IAM, organisation policies, the Marketplace order) are Google’s; problems inside the Databricks software (the workspace, Spark, Unity Catalog, SQL warehouses, jobs, serving) are Databricks’. A cluster that will not start is usually the first kind, which is why the troubleshooting section below reads the error code before anyone opens a ticket.

What are Google Cloud quotas, and why do they matter here?

A quota is permission, not capacity: it caps how much one project may use, and it does not promise that the machines exist. On classic compute, your project’s quotas are what cap Acme’s clusters.

A credit card limit is the closest everyday picture. The bank lets you spend up to, say, £5,000, but that does not mean the shop has the sofa in stock. And if the bank gives you one limit for travel and another for groceries, spare grocery limit does not pay for a flight. The card limit is the quota, the shop’s stock is a zone’s capacity, and the separate limits are the per-family CPU pools such as CPUS and N2_CPUS described below.

A quota is a limit on how much of a shared resource one project may use. Google’s Cloud Quotas overview splits them into allocation quotas (how many of something you can hold at once, such as CPUs or disk) and rate quotas (how many API calls per minute), and separates both from system limits, which are fixed and cannot be changed. Usage in one project does not affect another project’s quota.

How does this apply to Acme? The reason quotas matter for Databricks is the split from Part 1: classic compute runs as Compute Engine VMs in your project, so your quotas cap how many nodes Acme can run. Serverless compute runs in Databricks’ projects and does not use them. Four details of Compute Engine quotas cause most surprises, all from Google’s Compute Engine quota page:

CPU quota is regional and split by family. E2 and N1 share the CPUS pool; N2 uses its own N2_CPUS pool; newer families such as N4 use CPUS_PER_VM_FAMILY with a vm_family dimension. Some new projects also have a global CPUs (All Regions) quota that caps every region at once.
Preemptible quota changes the rules once granted. Spot VMs (Google’s spare-capacity VMs, which Databricks calls preemptible) consume standard quota until you request preemptible quota (PREEMPTIBLE_CPUS). After it is granted in a region, Spot VMs can only consume preemptible quota and cannot go back.
Local SSD quota moved. The old LOCAL_SSD_TOTAL_GB quota is deprecated; local SSD now counts against LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY.
A quota is not capacity. Quota is permission to use up to an amount. A zone can still run out of a machine type, which surfaces as a stockout. Guaranteed capacity comes from Compute Engine reservations, and the Databricks project role (Part 2) includes permission to list reservations.

Which quotas does a workspace need?

Databricks publishes the quotas it needs on its resource quotas page. The Google defaults are enough to start and test; the “at scale” column is its recommendation for production. I have replaced the page’s field-name column, which has copy errors (it lists the CPU metric against several disk rows), with the quota names from Google’s own page.

the Google Cloud quotas Databricks publishes for a workspace project, per region: default values are enough to start, and for running at scale it recommends 2,400 CPUs, 300 N2 CPUs, 500 in-use IP addresses, 50 TB each of Persistent Disk Standard, Persistent Disk SSD and Local SSD, 500 instance groups and managed instance groups, 300 routes, 275 subnetworks, 6,000 Monitoring ingestion requests a minute and 100 service accounts; classic and pro SQL warehouses need their own N2 CPU, Persistent Disk SSD and Local SSD headroom

Two notes on that table. First, Databricks names N2 because it is a common default; if Acme standardises on N2D or C4A instead, it raises that family’s quota. Second, classic and pro SQL warehouses run in your project and will not start without their own N2 CPU and disk headroom, while serverless SQL warehouses do not touch these quotas. The Databricks page also asks you to wait 15 minutes after any quota change before creating a workspace.

How does Acme size quota for month-end?

Quota planning is arithmetic on the workload’s shape. Acme’s busiest hour is the month-end close, when the nightly pipeline, a backfill and the analysts’ classic clusters overlap. The supply team estimates 25 clusters at once, each one driver and 12 workers on n2-highmem-8, with Spot workers that fall back to on-demand, which is the PREEMPTIBLE_WITH_FALLBACK_GCP setting from Part 1.

turning Acme's month-end peak into quota requests: 25 concurrent clusters of one driver and 12 workers on n2-highmem-8 give 325 nodes and 2,600 vCPUs; with Spot workers and preemptible quota granted, the 300 workers draw 2,400 PREEMPTIBLE_CPUS and the 25 drivers 200 N2_CPUS, but fallback to on-demand can push all 2,600 onto N2_CPUS, so Acme requests 3,000; the 325 nodes need 650 IP addresses, which fits the /22 subnet; and 325 nodes with 180 GB of persistent disk each need about 59 TB of disk quota

The numbers work out as follows:

CPUs. 25 clusters times 13 nodes is 325 nodes; times 8 vCPUs is 2,600 vCPUs. Because Acme has been granted preemptible quota, the 300 Spot workers draw 2,400 from PREEMPTIBLE_CPUS and the 25 on-demand drivers draw 200 from N2_CPUS. But fallback means that on a day with no spare capacity, every worker can land on on-demand VMs, so all 2,600 could hit N2_CPUS. Acme requests 3,000 there for headroom, and 2,400 or more of preemptible CPUs.
IP addresses. Each node uses two IP addresses (Part 1 explains why), so 325 nodes need 650. Acme’s subnet dbx-prod-euw2 is a /22, which holds about 500 nodes. It fits. Part 3 covers subnet sizing in full.
Disk. Part 1 shows each node with a 30 GB boot disk and a 150 GB container root volume, so 325 nodes hold about 59 TB of persistent disk, counted against the quota of the disk type in use. Local SSD adds 375 GB per disk on top, against the per-family local SSD quota.

Size for the autoscaling maximum, not the average, and add any idle pool VMs, because both hold CPUs and IP addresses whether or not work is running.

To check usage against limits and request an increase from the command line, Google provides gcloud compute regions describe and the Cloud Quotas commands. The quota ID for regional CPUs is CPUS-per-project-region; list the IDs in your project before you request, because each family has its own. I have not run these:

# Regional quotas in the workspace project: metric, usage, limit
gcloud compute regions describe europe-west2 --project=acme-dbx-prod \
  --flatten=quotas --format="table(quotas.metric,quotas.usage,quotas.limit)"

# Find the quota IDs for N2 CPUs (Cloud Quotas API)
gcloud quotas info list --service=compute.googleapis.com \
  --project=acme-dbx-prod --filter="quotaId~N2"

# Ask for 3,000 N2 CPUs in europe-west2 (use the quota ID the list returned)
gcloud quotas preferences create --service=compute.googleapis.com \
  --project=acme-dbx-prod --quota-id=N2-CPUS-per-project-region \
  --preferred-value=3000 --dimensions=region=europe-west2 \
  --email=platform-team@acme.example --justification="Databricks month-end peak"

Then set an alert. On the IAM & Admin, Quotas & System Limits page, the menu on a quota row has Create usage alert, which Google’s quota alerts page describes as a notification when usage reaches a percentage of the limit, sent by email, SMS or Pub/Sub. Acme sets 80% on CPUs, N2 CPUs, preemptible CPUs and disk in both workspace projects.

Which Databricks limits should you know?

The second kind of ceiling belongs to Databricks. Its resource limits page lists hundreds; these are the ones I see matter in Google Cloud deployments.

Databricks resource limits on Google Cloud that most often matter, as of October 2026: 20 workspaces per account, 2,000 tasks running at once and 10,000 jobs created per hour per workspace, 12,000 saved jobs, 1,000 SQL warehouses and 1,000 secret scopes per workspace, 1,000 IP access list CIDR values, 10,000 users and service principals, 1,000 storage and service credentials and 10,000 external locations per metastore, a million tables per metastore, 20 network connectivity configurations per region, and Jobs API rates of 20 and 35 requests a second; fixed limits cannot be raised, the others can through the account team

Read the last column carefully. A limit marked fixed cannot be raised; the others can, through your Databricks account team. Two of these shape design. Twenty workspaces per account is another reason, alongside Unity Catalog’s one metastore per region, to plan few workspaces, as Part 1 recommends. And the Jobs API rates (20 run-now calls a second, 35 runs/submit calls a second, per workspace) matter if an orchestrator outside Databricks fires many jobs at once: back off and retry instead of looping.

What are the three kinds of ceiling?

Putting the two kinds together with a third gives a simple way to plan: know who owns each limit, and how it is lifted.

Picture a restaurant in a rented building. The landlord caps how much power you may draw, and raises the cap if you ask on the right form. The franchise caps how many branches you may open, lifts some of its caps on request and treats others as fixed rules. And the dining room is the size you built it: no form changes that, only building work. The landlord is Google Cloud and its quotas, the franchise is Databricks and its resource limits, and the dining room is a design-time ceiling such as a subnet’s size.

three kinds of ceiling and who owns each: Google Cloud quotas belong to your project and are raised from the Quotas and System Limits page or the Cloud Quotas API; Databricks resource limits belong to the Databricks account and workspaces and are raised through the account team unless fixed; design-time ceilings such as subnet size, Private Service Connect endpoints per host project and Cloud NAT ports per VM belong to your architecture and can only be fixed by redesign, so size them early

The third kind, design-time ceilings, is the one that bites, because no ticket raises it. A workspace’s subnet size is set by your network design (Part 3 covers moving to a bigger subnet). The Private Service Connect page allows up to ten endpoints of each type per region per host project, shared by every workspace in that VPC. And Cloud NAT gives each VM a minimum of 64 source ports by default with static port allocation (32 with dynamic allocation), which a Spark job that opens many outbound connections can exhaust. Part 3 owns the fixes; this post’s job is to make sure someone sizes them before go-live.

How does this apply to Acme? Acme’s 3,000 N2_CPUS are the landlord’s cap, raised with the quota request above. Its allowance of 20 workspaces per account is the franchise’s cap. And the size of the subnet dbx-prod-euw2, which has to hold 650 IP addresses at month-end, is the dining room: only a network change moves it.

How do you keep costs down on both bills?

Cut the Databricks bill by choosing the right compute type and capping it; cut the Google Cloud bill with Spot VMs, commitments and keeping data close. Serverless has only the first bill. In the sofa picture from the billing section, you can haggle with the driver and with the van firm separately, but the removal firm gives you only one price to negotiate.

There are two bills to optimise, and the levers are different. Everything on the left of this diagram applies to classic compute only, because serverless has no Compute Engine line.

cost levers on both bills for Acme: on the Google Cloud bill, Spot workers with an on-demand driver, committed use discounts for the steady baseline, the right machine family, data kept in region and auto-termination; on the Databricks bill, jobs compute for production, SQL warehouses for SQL, serverless for spiky work, compute policies that cap DBUs per hour and force auto-termination; and to make costs visible, custom tags that become Compute Engine labels in the Cloud Billing export, the billing system tables and Databricks budgets

On the Google Cloud bill:

Spot workers. Spot VMs are up to 91% cheaper than on-demand for many machine types, and Compute Engine can reclaim them at any time. Keep the driver on-demand and use fallback, as Part 1 configures.
Committed use discounts (CUDs). A committed use discount is a lower price in return for a one or three year commitment. Google’s CUD overview describes resource-based CUDs for a region and machine configuration, and spend-based compute flexible CUDs. They suit the steady baseline of classic clusters, and they do nothing for serverless.
Machine family. Part 1 covers the choice; the short version is to measure, and to check Arm builds of every library before moving to C4A.
Keep data close. Buckets in europe-west2 next to the clusters avoid inter-region transfer, and Google’s Cloud NAT overview notes that traffic to Google APIs goes through Private Google Access even when a VM also uses Cloud NAT.

On the Databricks bill:

Jobs compute for production. The Databricks cost optimisation best practices say non-interactive workloads “will cost significantly less” on jobs compute than on all-purpose compute, and that a SQL warehouse is the most cost-efficient engine for interactive SQL.
Serverless for spiky or interactive work, where idle clusters would otherwise burn DBUs and VMs.
Compute policies. A compute policy limits what users may create. The policy reference includes dbus_per_hour, a calculated attribute for the maximum DBUs a cluster can use per hour, as well as autotermination_minutes and custom_tags.*. Acme’s policy for classic job clusters looks like this:

{
  "dbus_per_hour": { "type": "range", "maxValue": 60 },
  "autotermination_minutes": { "type": "fixed", "value": 30, "hidden": true },
  "custom_tags.team": { "type": "fixed", "value": "supply" },
  "gcp_attributes.availability": { "type": "fixed", "value": "PREEMPTIBLE_WITH_FALLBACK_GCP" }
}

Photon, the native query engine, has its own DBU rate; test whether it lowers the total for your workloads before making it the default. Part 7 covers it.
Background features have costs too. Predictive optimisation, data quality monitoring and default storage each appear as their own billing_origin_product in the usage table. Watch them.

Which lever first? This is my order, cheapest to pull first, and Acme’s policy above already pulls three of them:

1. Production running on all-purpose compute?        -> move it to jobs compute
2. Clusters idle for long stretches?                  -> auto-termination, or serverless
3. Workers can tolerate being reclaimed?              -> Spot workers, on-demand driver
4. A team keeps surprising you?                       -> compute policy with dbus_per_hour
5. A steady baseline for a year or more?              -> committed use discounts
Only then: benchmark machine families or Photon, which cost test time to prove.

How do you make cost visible?

Visibility comes from tags (key and value pairs you attach to compute). The tagging guide says custom tags go to two places: the custom_tags column of system.billing.usage, and labels on the Compute Engine VMs and persistent disks. That second path means the same tag shows up in the Cloud Billing export to BigQuery, so Acme’s finance team can split both bills by team. Google’s label rules apply on the way: keys and values become lowercase, are limited to 63 characters, and at most 54 custom tags propagate. For serverless, tags come from serverless usage policies, in Public Preview as of October 2026.

Finally, budgets. In the account console under Usage, Budgets, an account admin can set a monthly threshold for the whole account or filtered by workspace and tag, with email recipients. The budgets page notes that budgets are measured in US dollars at list price, so they are an early warning, not an invoice.

What does day-two operations look like?

Day-two operations is the work after go-live: watching, alerting, fixing and changing the platform without breaking it. For Acme it is a loop.

Acme's day-two operations loop: signals come from the billing and compute system tables, cluster event logs kept for 60 days, Cloud Logging admin activity for the workspace service account, quota usage on the Quotas and System Limits page and Databricks budgets; they feed a cost dashboard, SQL alerts and Cloud Monitoring quota alerts; those trigger actions such as a compute policy change, a quota increase request, a bundle redeploy or a support case, and the changes show up in the signals again

The signals, and where they live:

Billing and compute system tables. Besides system.billing.*, the compute system tables include system.compute.clusters (every cluster configuration over time), system.compute.node_timeline (per-minute utilisation of each node) and system.compute.node_types. Databricks recommends AI/BI dashboards on the billing tables, and account admins can import a pre-built cost dashboard. Audit logs in system.access.audit belong to Part 6.
Cluster event logs. The compute management page says event logs capture lifecycle events such as creation and termination, are kept for 60 days, and can be downloaded. It also notes that terminated compute is deleted 30 days after termination unless pinned.
Google Cloud’s side. Cloud Logging’s Admin Activity audit logs in acme-dbx-prod, filtered on the workspace service account from Part 2, show the Compute Engine calls Databricks made on your behalf and their errors. The Quotas page shows usage against limits.

Change follows the same discipline as any software. Acme deploys jobs and pipelines with Declarative Automation Bundles (formerly Databricks Asset Bundles): YAML files in Git that describe Databricks resources and the targets to deploy them to. A trimmed databricks.yml (I have not run this one):

bundle:
  name: acme-supply
resources:
  jobs:
    nightly_supply_build:
      name: nightly_supply_build
      tags: { team: supply }
targets:
  dev:
    mode: development
    workspace: { host: https://<acme-dev-workspace>.gcp.databricks.com }
  prod:
    mode: production
    workspace: { host: https://<acme-prod-workspace>.gcp.databricks.com }

The pipeline that runs databricks bundle deploy -t prod authenticates without keys, with a Google service account; Part 2 shows that identity chain, so I do not repeat it here. Bundles deploy what runs inside a workspace; the workspaces themselves, and the projects, networks and keys under them, come from Terraform in the same kind of pipeline, and Part 2 lists which Terraform resource does each step.

Why did my cluster not start?

Read the error code in the cluster event log before you open a ticket: it usually names the cause and the side that owns it. When a car will not start, the dashboard light tells you whether you are out of fuel, have a flat battery or have a fault in the engine, and that decides whether you walk to the petrol station or call the garage. The event log’s termination code is that dashboard light. The trap is opening a Databricks ticket for what is really a Google Cloud quota.

Most classic launch failures on Google Cloud are Google Cloud problems surfacing through Databricks, and the cluster event log says which one. Databricks publishes a page of termination error codes with causes and fixes; these are the ones I would expect Acme to meet.

classic cluster launch failures on Google Cloud by termination error code: quota exceeded, insufficient zonal capacity, stockout from a narrow configuration, subnet IP space exhausted, a missing resource, the relay tunnel failing, a bootstrap timeout, Spot workers reclaimed and rate limiting, each with what it means and the first fix to try

Three of them deserve a sentence each. GCP_RESOURCE_QUOTA_EXCEEDED quotes Compute Engine’s own quota name, for example SSD_TOTAL_GB or LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY, so the fix is a quota request for exactly that name and region. GCP_INSUFFICIENT_CAPACITY is a stockout, the “does not have enough resources available” error Google documents as ZONE_RESOURCE_POOL_EXHAUSTED; Databricks’ answer is flexible node types, which let a cluster fall back to compatible instance types (not for GPUs). And GCP_IP_SPACE_EXHAUSTED means the subnet is full; the error page suggests expanding or replacing the subnet, and Part 3 explains why Databricks documents only the new-subnet route.

What can go wrong?

Cancelling the subscription deletes every workspace. The subscription documentation warns that users with the consumerprocurement.orders.cancel permission on the billing account can cancel it, and that cancellation “deletes all workspaces in the Databricks account”. Check who holds Billing Administrator on that billing account.
Expecting Google Cloud discounts on serverless. Google Cloud credits and discounts can apply to resources in your own project. The documentation says they do not apply to serverless compute.
Quota in the wrong family. Plenty of CPUS quota does not help an N2 cluster, and preemptible quota, once granted, is the only pool Spot VMs can use.
Sizing for the average. Autoscaling maximums, fallback to on-demand and idle pools all take quota and IP addresses at the worst moment.
Reading list prices as costs. list_prices and budgets use list prices; your invoice reflects your contract.
Choosing the plan late. Moving to Private Service Connect later means an Enterprise subscription and a network change, not a toggle.
A billing account change at the last minute. Entitlement transfers need the vendor’s agreement, are not guaranteed and can take a week or more.

What would I do now?

List who can cancel. Run gcloud billing accounts get-iam-policy on the billing account behind the Databricks order, check Organisation Owners, cut both lists to named people, and record the Google order number.
Decide the plan from requirements. If production needs Private Service Connect or customer-managed keys, start on Enterprise.
Turn the peak into quota requests. Peak nodes times vCPUs, per family, standard and preemptible; peak nodes times two for IP addresses; then raise the quotas in both workspace projects and set 80% alerts.
Tag everything from day one. A compute policy that fixes custom_tags.team and caps dbus_per_hour, serverless usage policies for serverless, and the Cloud Billing export to BigQuery switched on.
Write the support runbook. Google Cloud Customer Care first, a Databricks support contract if you need one, and a rule that every ticket quotes the event log error code.

What did we learn?

Databricks on Google Cloud has two plans as of October 2026, Premium and Enterprise, and the plan decides architecture: customer-managed VPC needs Premium, while Private Service Connect and customer-managed keys need Enterprise.
Databricks bills in DBUs per SKU, metered per second, and everything is paid on one Google Cloud bill; classic compute adds a Compute Engine line in your project, while serverless folds the machines into the DBU price, where your Compute Engine discounts do not apply.
Subscribing creates a Marketplace order, an account and an entitlement, then the Databricks account; changing the billing account is an entitlement transfer, and anyone with consumerprocurement.orders.cancel can delete every workspace by cancelling the order.
Classic clusters are capped by three kinds of ceiling: Google Cloud quotas in your project (regional, per family, with a separate preemptible pool), Databricks resource limits, and design-time ceilings such as subnet size that only redesign can lift.
Cost control works on both bills, with Spot workers and committed use discounts on one and jobs compute, serverless and compute policies on the other, and tags that flow into both the billing system tables and the Cloud Billing export.
Day-two operations is a loop of signals, alerts and actions, and a failed cluster is diagnosed by its termination error code before anyone opens a ticket, starting with Google Cloud Customer Care.

What comes next?

The last part, Databricks on Google Cloud, Part 10: Migration to Google Cloud, moves Databricks workloads from AWS or Azure to Google Cloud. It builds on this part directly: plan differences between clouds change the commercial case, and quotas, limits and the operating routine all have to be in place before the first migrated job runs.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional.