Databricks on Google Cloud, Part 1: Foundations, how Databricks is built on Google Cloud

Databricks runs on all three large clouds. On Google Cloud you buy it through Google Cloud Marketplace and pay for it on your Google Cloud bill. The notebooks, Spark and Unity Catalog look the same wherever you run them. What changes is everything underneath: which project the virtual machines land in, who sends the bill, which identity creates the VMs, and which Google Cloud policy quietly blocks your first deployment.

That underneath part is what this post is about. In the three prerequisites we covered what a lakehouse is, what Google Cloud is and the identity and security basics, such as tokens, single sign-on and keys. Now we put them together, starting with what Databricks itself is. By the end you should be able to draw a Databricks deployment on Google Cloud from memory, name every resource it creates in your project, and explain why a cluster failed to start.

I will use the same company throughout the series: Acme Consumer Brands, the fictional household products company from the Production-ready agents series. Acme’s supply team runs Databricks on Google Cloud next to BigQuery, which the marketing team uses, and the agents in that series, published to Gemini Enterprise, read data from both. In this post Acme sets up its first two workspaces, one for production and one for development.

Acme's estate in one picture: agents in Gemini Enterprise read marketing data in BigQuery and supply data in Databricks on Google Cloud; this post builds the Databricks side, with a production and a development workspace

The series overview describes the running example, and the agents series’ business and data prerequisite explains both tables.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: where Databricks actually runs on Google Cloud, and what it creates in your project, in three minutes. The narration is a synthetic voice; the script and animation are mine.

What are the key ideas?

Six terms carry the whole post. If you know Databricks on AWS or Azure, the first four will be familiar and only their Google Cloud landing points are new.

Databricks account. The top-level Databricks tenant (the organisation-wide container) where you manage users, workspaces, metastores and network settings. On Google Cloud it is created when you subscribe in Marketplace.
Workspace. The regional environment where people run notebooks, jobs, SQL and machine learning. On Google Cloud, a classic workspace is tied to exactly one workspace project (the Google Cloud project that holds its compute and storage) and one region.
Control plane. The back-end services Databricks runs for you: the web application, REST APIs, the cluster manager, the job scheduler and Unity Catalog. It lives in Databricks’ cloud account, not yours.
Classic compute plane. The network and VMs in your Google Cloud project where classic clusters run.
Serverless compute plane. Compute that Databricks runs on your behalf in Databricks-managed Google Cloud projects, in the same region as your workspace.
Workspace storage. The Cloud Storage buckets that hold a workspace’s internal data. In a classic workspace they sit in your workspace project.

The diagram below maps each Databricks construct to the Google Cloud resource it lands on. Keep it in mind as you read: nearly every design question on Google Cloud turns into which project, and which region?

how Databricks constructs land on Google Cloud: the account maps to a Marketplace order and billing account, a workspace to one project and one region, the classic compute plane to a VPC subnet, cluster nodes to Compute Engine VMs and workspace storage to Cloud Storage buckets

What is Databricks?

The simple definition: Databricks is a managed platform for building a lakehouse and everything that runs on it, from data pipelines to dashboards to AI agents.

The formal definition, from the documentation: “Databricks is a unified, open analytics platform for building, deploying, sharing, and maintaining enterprise-grade data, analytics, and AI solutions at scale.” The documentation calls the whole product the Databricks Data + AI Platform, and says it “integrates with cloud storage and security in your cloud account, and manages and deploys cloud infrastructure for you”. On Google Cloud, some of that infrastructure ends up in your own project.

The two diagrams below show every product, by its October 2026 name, and what Acme uses it for. A few of them need a sentence more here; Part 7 covers each product in depth.

Workspace: the web environment where people work, with notebooks (documents that mix code, results and text), files, Git folders (a copy of a Git repository inside the workspace, so notebooks and code are version controlled) and the SQL editor.
Classic compute: a cluster of Compute Engine virtual machines (VMs) in your own project. An all-purpose cluster is shared for interactive work; a job cluster exists for one job run. Choose classic when you need control over machine types, networking or libraries.
Serverless compute: compute that Databricks runs and scales in its own account. Databricks recommends it for almost all pipelines, and new notebooks use it by default.
SQL warehouses: compute sized for SQL, in three types (classic, pro and serverless). Databricks recommends serverless where it is available.
Unity Catalog: “the unified governance layer for data and AI built into Databricks”. It names every table as catalog.schema.table, controls who may use it, records lineage and logs access for audit. Part 6 covers it in depth.

Lakeflow (ingestion, pipelines and jobs), Genie, Agent Bricks, Databricks Apps and Lakebase appear in the table below, and Part 7 explains each one, including former names such as Delta Live Tables.

the Databricks Data + AI Platform as layers: workloads for ingesting, analysing, AI and apps on top; classic compute, serverless compute and SQL warehouses running the Databricks Runtime in the middle; Unity Catalog governing everything; and your data as Delta and Iceberg tables in Cloud Storage at the bottom

the main Databricks products in one table: what each one is, and what the supply team at Acme Consumer Brands uses it for

How is the platform laid out on Google Cloud?

This is the most important idea in the post. The Databricks architecture documentation splits the platform into a control plane and a compute plane. The names describe the jobs: the control plane controls (it decides what runs, where and when), and the compute plane computes (it is where your data is actually read and processed).

The bottom line: the control plane decides, the compute plane does the work, and with classic compute the work happens in your own Google Cloud project. The trap is to assume that a managed service runs everything in the vendor’s account. On Google Cloud, classic cluster VMs, their disks and their network traffic are yours.

A simple example makes the split easy to picture. Think of a catering company hired for a dinner at your house:

The caterer’s office takes the booking, plans the menu, sends instructions to the cooks and keeps the paperwork. It never touches your food. That is the control plane.
The cooking happens in your own kitchen, on your cooker, and the gas and electricity appear on your bill. That is the classic compute plane.
For some events the caterer cooks in its own kitchen nearby and brings the dishes over: quicker to arrange and nothing to set up at home, but you no longer choose the cooker. That is the serverless compute plane.
Either way, the ingredients come from your fridge, with a key you handed over. That is your data in your own Cloud Storage buckets.

The control plane holds “the backend services that Databricks manages in your Databricks account”, including the web application. It runs in Databricks-managed Google Cloud projects, never in yours. It serves the web application and REST APIs, creates and resizes clusters, schedules jobs, routes SQL queries and hosts Unity Catalog.

The compute plane is “where your data is processed”, and there are two kinds.

The classic compute plane runs in your Google Cloud project: new compute resources are created “within each workspace’s virtual network in the customer’s Google Cloud account”. A virtual network (on Google Cloud, a VPC, virtual private cloud) is a private network, defined in software, that controls which IP addresses machines get and what they may reach. The VMs, disks and network traffic appear in your project and on your Google Cloud bill.
The serverless compute plane runs in Databricks’ account, “in the same Google Cloud region as your workspace’s classic compute plane”. Serverless SQL warehouses, serverless notebooks and jobs, and serverless Lakeflow pipelines run there, and Databricks isolates each workspace’s serverless compute inside its own network boundary.

How does this apply to Acme? The same three pieces map one to one. The caterer’s office is the Databricks control plane, running the web application, the job scheduler, the cluster manager and Unity Catalog in Databricks’ account. Cooking at home is the supply team’s nightly pipeline: its cluster VMs start in acme-dbx-prod, inside Acme’s VPC, and appear on Acme’s Google Cloud bill. The caterer’s own kitchen is the serverless SQL warehouses behind the analysts’ dashboards, in the same region, europe-west2. And the ingredients, the supply data, stay in Acme’s own buckets in both cases. The last section explains why Acme splits the work this way.

Here is a classic workspace in detail, where the compute and the workspace buckets are in your project:

a classic workspace on Google Cloud: the control plane with the web app, REST APIs, job scheduler, cluster manager and notebook and query metadata is in Databricks' account; your project holds the VPC with the Compute Engine driver and worker VMs, the three workspace storage buckets created with the workspace, and your own data buckets; a request goes from the user to the control plane, which starts VMs in your project that read your data and return the result

The classic compute plane runs in your workspace project, in your VPC. Three Google Cloud features make it work, and a fourth makes it private:

Secure cluster connectivity (SCC). Cluster nodes have no public IP addresses and no open inbound ports. Each cluster opens an outbound connection to the SCC relay in the control plane, and the control plane sends its instructions back through that tunnel. It is the default for every workspace.
Cloud NAT. Because nodes have no public IPs, they need network address translation (NAT) (a service that lets private VMs make outbound connections through shared public addresses) to reach the control plane over the internet. Cloud NAT is Google’s managed NAT.
Private Google Access (PGA). A subnet setting that lets VMs with only internal IP addresses reach Google APIs such as Cloud Storage. Databricks nodes rely on it to read and write the workspace buckets and your data.
Private Service Connect (PSC), optional. PSC gives a service a private endpoint: an internal IP address inside your own VPC. With classic compute plane PSC (also called back-end PSC), the cluster reaches the control plane through two such endpoints in your VPC instead of over the internet through Cloud NAT: a workspace endpoint for the Databricks REST APIs and an SCC relay endpoint for the secure cluster connectivity relay. Inbound PSC (also called front-end PSC) is the matching path for people, so browsers and API clients reach the web application privately. The Private Service Connect page lists two requirements: “The workspace is on the Enterprise plan”, and “You must use a customer-managed VPC”. Part 3 covers PSC in depth, including the DNS, the private access settings and why a PSC workspace may still keep Cloud NAT.

The serverless compute plane runs in Databricks’ account. A serverless workspace goes one step further: it uses only serverless compute and Databricks-managed default storage, so nothing at all is created in your project apart from the buckets you choose to connect. Serverless compute reaches those buckets through a Unity Catalog storage credential and external location (Part 5 explains both), and to reach resources through private endpoints, account admins attach a network connectivity configuration (NCC), which the documentation describes as “account-level regional constructs that manage private endpoint creation”. Part 4 covers serverless networking.

a serverless workspace on Google Cloud: the control plane, the serverless compute plane inside a network boundary for the workspace in the same region, and the default storage that holds workspace data and the default catalog all live in Databricks' account; your project holds only your own buckets, which serverless compute reaches through a Unity Catalog storage credential and external location

How do accounts, workspaces and metastores fit together?

The bottom line: one account for the company, a workspace for each place a team works, and one metastore per region, shared by every workspace in that region. The common mistake is to picture a metastore per workspace, as there was a Hive metastore per workspace before Unity Catalog.

A simple example: a company with several offices in one city. Head office keeps the staff list and pays the bills. Each office is where a team actually works. The city’s records room holds the files, and every office in the city looks into the same room, each seeing only the drawers it holds keys for. Head office is the account, an office is a workspace, the records room is the metastore, and the drawer keys are Unity Catalog grants.

The Databricks account is the top level. It holds identities (users, groups and service principals, which are identities for automation), workspaces, Unity Catalog metastores and usage. On Google Cloud, you create it by subscribing to Databricks in Google Cloud Marketplace, as the next section shows.
A workspace is one regional environment where a team works. Each classic workspace runs its compute in a Google Cloud project that you choose when you create it.
A Unity Catalog metastore is the top container for governed data in one region. You can attach one metastore to many workspaces in the same region, so they all see the same tables.

Databricks supports 15 Google Cloud regions, including europe-west2. Acme chose europe-west2 (London) for its supply workspaces. That choice matters more than it looks: some features are only available in some regions, and the Databricks regional table shows, for example, that custom agents and agent evaluation are available in europe-west2 but not in europe-west1 (Belgium) as of October 2026.

How does this apply to Acme? Head office is Acme’s one Databricks account, which holds its users, groups and service principals. Its two offices are the production and development workspaces, both in europe-west2, and both look into the same records room: the one Unity Catalog metastore for europe-west2, as the diagram below shows. Production and development see different drawers because of grants, not because they have separate metastores.

how the Databricks objects map to Google Cloud for Acme: a Google Cloud billing account pays for one Databricks account bought through Google Cloud Marketplace; the account holds identities, a Unity Catalog metastore for europe-west2 and two workspaces, each in its own Google Cloud project in that region

Google Cloud Marketplace is Google’s catalogue of third-party software that you buy and pay for through your Google Cloud bill. Databricks is listed there, and subscribing is how a Databricks account on Google Cloud comes into existence.

To subscribe you need a Cloud Billing account (the Google object that pays for usage) and two IAM roles: Billing Administrator (roles/billing.admin) on that billing account, and Viewer (roles/viewer) on a project linked to it. In Marketplace you pick that project in the project picker, click Subscribe, choose the billing account, register with Databricks and then click Manage on Provider to activate the subscription. You land in the Databricks account console at accounts.gcp.databricks.com. The person who subscribed becomes the account owner and can add other account admins. Companies with a negotiated contract do not use this self-service path: they get a Marketplace private offer (a custom price and terms for one buyer) from their Databricks account team.

The billing model is simple once you see it, and it is the part people most often get wrong:

one Marketplace order links the Databricks account to one Cloud Billing account: DBU usage from every workspace is metered by Databricks and charged through that order, while the VMs, disks, buckets and NAT in each workspace project are billed by Google to that project's own billing account

One order links one subscription to one billing account. The subscription creates a Google order number, which you can see in the account console under Settings, Subscription and Billing, and match in Marketplace under Manage orders. Most organisations have one active order; large ones can have more than one.
Databricks usage is charged through that order. Databricks meters usage in Databricks Units (DBUs), a normalised unit of processing per hour. How many DBUs a workload uses depends on the compute type (for example jobs versus all-purpose) and the machine type. The charges go to the billing account behind the order.
Infrastructure bills to each workspace project. The VMs, disks and buckets in each classic workspace project are ordinary Google Cloud resources, so Google bills them to whatever billing account that project is linked to. The project you picked in Marketplace only chose the billing account; workspaces can go into any number of other projects.

The caterer from earlier splits its bill the same way. It sends one invoice for its service, whichever house it cooked in: that is the DBU charge on the Marketplace order. The gas and electricity each kitchen used go on that house’s own utility bill: that is the VMs, disks and buckets billed to each workspace project’s billing account. The misconception to drop is that the project you picked in Marketplace is where workspaces must live, or that it pays for their VMs. It only chose the billing account for the order.

How does this apply to Acme? The DBUs from both Acme workspaces arrive through Acme’s Marketplace order. The VMs, disks and buckets in acme-dbx-prod are billed to whichever billing account acme-dbx-prod is linked to, and those in acme-dbx-dev to whichever account acme-dbx-dev is linked to. If the two projects were linked to different billing accounts, the infrastructure would land on two bills while the DBUs stayed on one.

Two more facts belong here. First, the plan: as of October 2026 you choose between Premium and Enterprise. That choice matters for architecture, because a customer-managed VPC needs Premium and Private Service Connect needs Enterprise. Second, cancelling the subscription deletes every workspace in the account, so review on day one who is allowed to cancel it. Part 9 covers DBUs, the Marketplace order and entitlement, changing the billing account and that cancellation risk in full.

How do projects map to workspaces?

If you are new to Google Cloud, the resource hierarchy is the first thing to learn. An organisation (your company, tied to its Cloud Identity or Google Workspace domain) contains folders (groups of projects, usually by business unit or environment), and folders contain projects (the unit of billing, quota, enabled APIs and IAM, closest to an AWS account or an Azure subscription). Policies set high up are inherited by everything below. Prerequisite 2 covers the general idea; here is what Acme built:

Acme's Google Cloud resource hierarchy: the organisation holds a data-platform folder with the production and development workspace projects and a networking folder with the Shared VPC host project; IAM policies and organisation policies set at the top are inherited by every project below

A classic Databricks workspace is created inside one project, in one region, and both are fixed for its lifetime. The reverse is not true: one project can hold several workspaces, and the customer-managed VPC documentation says you can “configure multiple Databricks workspaces to share a single compute plane VPC”. So how many projects should you have?

A common answer is one project per workspace, and it is worth knowing where that habit comes from before copying it. Before Unity Catalog, permissions on data lived in each workspace’s own Hive metastore, so the only way to keep finance and supply apart was a separate workspace, and often a separate project and network. On Azure the pattern is even stronger, because every Azure Databricks workspace brings its own managed resource group. Carried over to Google Cloud at scale, it multiplies projects, and it runs into Shared VPC rules: the number of service projects per host project is a quota, and a service project can attach to only one host project, a limit that Google says cannot be increased.

Unity Catalog changes the starting point. Permissions on data now live at the account level, so one workspace can serve several teams, each seeing only its own catalogs.

A simple example: an office building. Before Unity Catalog the only lock was the front door, so every team that needed privacy got its own building. Unity Catalog adds locked filing cabinets inside, so teams can share a building and still see only their own files. You still put up a second building when the reason is the building itself: a different fire code, a different landlord or a separate electricity meter. Mapped back, buildings are workspaces and projects, filing cabinets are catalogs and grants, and the fire code, landlord and meter are organisation policies, security boundaries, quotas and billing.

My view, in this order:

Have fewer workspaces. Split workspaces by environment (production, development) and by real differences in security or network needs, not by team. Teams are separated by catalogs and grants.
Group workspaces into projects by owner and environment, for example one project for the supply team’s production workspaces, rather than one project each. Give a workspace its own project only when it needs a different security boundary, its own quotas or different organisation policies.
Keep one subnet per workspace, even in a shared project and VPC. A burst of clusters in one workspace cannot then use up the IP addresses another needs, firewall rules can differ per workspace, and a network mistake stays contained.
Attribute cost with tags, not projects. Databricks says that tags on compute “propagate to labels on GCE resources such as VM and its persistent disks”, so the Cloud Billing export can group VM and disk costs by a cost_center tag. A compute policy can make that tag mandatory. DBUs are recorded per workspace in system.billing.usage.

The same view as a quick test, cheapest answer first:

A team wants Databricks. What does it need?       Ask in order; stop at the first yes.
1. Same environment, same security needs?         -> no new workspace: a catalog and grants
2. A different environment, or real differences   -> a new workspace (and its own subnet) in the
   in network or security settings?                  existing project for that owner and environment
3. Different organisation policies, its own       -> only then a new project
   quotas or a different billing owner?

How does this apply to Acme? Acme starts small, with one production and one development workspace, so it uses acme-dbx-prod and acme-dbx-dev; when the marketing analytics team gets a production workspace, it will go into acme-dbx-prod on its own subnet, not into a new project.

The network can live in the same project or in a separate one:

Standalone VPC. The VPC sits in the workspace project. Simple, and fine for a single team.
Shared VPC. A host project owns the VPC and shares subnets with service projects (Google’s term for projects that use a host’s network). The workspace project is the service project. Network teams like this because they keep control of routing, NAT and firewall rules in one place. Do not confuse it with several workspaces sharing one VPC, which both models allow.

Acme chose Shared VPC: acme-net-host holds the VPC acme-dbx-vpc, Cloud Router and Cloud NAT, while acme-dbx-prod and acme-dbx-dev are service projects that each hold one workspace. The workspaces follow the naming pattern in the Databricks deployment guide, {organisation}-{environment}-{region}-{purpose}: acme-prod-euw2-supply and acme-dev-euw2-shared.

Three limits are worth knowing early. An account can hold 20 workspaces by default. You can create at most 200 workspaces per week in the same Google Cloud project; beyond that, creation fails with a message about creating a custom IAM role. And each workspace name must be 3 to 30 characters of letters, numbers, hyphens and underscores.

Where does a workspace land on each cloud?

If you know Databricks from Azure, one difference catches people out. A Google Cloud project is the equivalent of an Azure subscription or an AWS account, not of an Azure resource group (Prerequisite 2 lines the three clouds up). On Azure, Azure Databricks is a first-party Azure service: the workspace is itself an Azure resource in a resource group, and Databricks creates a second, managed resource group for the workspace’s VMs, network and storage, which you cannot edit directly. On Google Cloud and AWS, the workspace belongs to the Databricks account, and its classic compute runs in a project or AWS account that you choose.

where a Databricks workspace lands on each cloud: on Google Cloud you buy through Google Cloud Marketplace and the classic compute runs in a project you choose; on AWS the compute runs in an AWS account you choose, reached through a cross-account role; on Azure the workspace is itself an Azure resource in a resource group, with a managed resource group that Databricks controls

Which landing-zone settings should you check first?

A landing zone is the pre-built foundation of organisation, folders, projects, networks and policies that a platform team prepares before workloads arrive. Most failed first deployments I have seen on Google Cloud were not Databricks problems at all. They were landing-zone settings nobody checked.

Which organisation policies should you check?

The bottom line: IAM says who may act; an organisation policy says what nobody below it may do, however senior. That is why “I am Owner, so it cannot be a permissions problem” is the most common wrong turn in a first deployment.

A simple example: a hotel keycard opens your room, the gym and the pool. The hotel’s rule that the roof is closed to all guests overrides every card, so even the penthouse keycard does not open the roof door. The keycard is your IAM role; the hotel rule is the organisation policy.

An organisation policy is a guardrail set on the organisation, a folder or a project that restricts what anyone can do below it, whatever their IAM roles. It is the Google Cloud equivalent of an AWS service control policy or an Azure Policy assignment. These are the ones to check:

organisation policies to check before the first workspace: domain restricted sharing must allow Databricks' customer ID, resource locations must allow the workspace region, Shared VPC subnet restrictions must allow the workspace project, and a service account key ban affects some integrations

The first row catches the most teams. Domain restricted sharing limits IAM grants to principals from approved Google customer IDs. Databricks documents that you must add its customer ID, C01p0oudw, alongside your own before you provision a workspace, because the workspace service account that receives roles in your project belongs to Databricks’ organisation (Part 2 explains the steps). Since 3 May 2024, new organisations get this policy enforced by default as part of Google’s security baseline, so a brand-new organisation will hit it on the first attempt.

How does this apply to Acme? Suppose Priya, Acme’s platform lead (Part 2 introduces her properly), holds Owner on acme-dbx-prod: the penthouse keycard. If Acme’s organisation enforced domain restricted sharing without C01p0oudw on the list, her first workspace creation would still fail, because the policy blocks the grant to Databricks’ service account whoever makes it. Checking the effective policies first takes a minute.

You can check the effective policy on the workspace project with gcloud (not run here; the commands follow the public gcloud reference):

# Which customer IDs may receive IAM roles in this project?
gcloud org-policies describe iam.allowedPolicyMemberDomains \
  --effective --project=acme-dbx-prod

# Which regions may resources be created in?
gcloud org-policies describe gcp.resourceLocations \
  --effective --project=acme-dbx-prod

Which quotas and APIs does a workspace need?

A quota is a limit Google Cloud places on how much of a resource a project may use. Compute Engine CPU quotas are regional (any zone in the region draws from the same pool), and many machine families have their own separate pool: N2 uses N2_CPUS, while newer families such as C4A use CPUS_PER_VM_FAMILY with a vm_family dimension. A project can have plenty of general CPU quota and still fail to start an N2 cluster.

A simple example: a car hire branch allowed to lend 100 saloon cars and, under a separate allowance, 10 vans. A customer who asks for 12 vans is turned away while 80 saloons sit in the yard. The saloon allowance is the general CPU quota; the van allowance is a family pool such as N2_CPUS.

How does this apply to Acme? Acme’s nightly supply job cluster (shown later in this post) uses n2-highmem-8 nodes and can grow to 16 workers plus the driver: 17 nodes of 8 vCPUs, or 136 N2 vCPUs at its peak. That number has to fit in N2_CPUS for europe-west2 in acme-dbx-prod, next to everything else running there, whatever the general CPU quota says.

Databricks publishes the quotas it needs. The defaults are enough to start; for running at scale, and for SQL warehouses, which need their own headroom, it recommends much higher CPU, IP address and disk quotas, which Part 9 lists in full. After any quota change, wait 15 minutes before creating the workspace.

During creation Databricks enables two APIs on the workspace project if they are off, the Compute Engine API and the Cloud Storage API, and it does not disable them when a workspace is deleted.

Which decisions can you not undo later?

Some choices are fixed when the workspace is created, so settle them before the first one:

The workspace project and region. Workspace configuration cannot be changed after creation.
The VPC. You cannot move a workspace from a Databricks-managed VPC to your own, and you cannot switch it to a different customer-managed VPC later. You can move it to a new subnet in the same VPC. The Databricks page Update workspace network configuration says: “To change the IP range used by your workspace, you must create a new subnet with the desired range, create a new network configuration object, and then update your workspace to use this new network configuration.”

Can you instead grow the subnet you already have, say from /29 to /22? Google Cloud itself allows it: you can expand a subnet’s primary IPv4 range in place by giving it a shorter prefix, as long as the larger range does not overlap another subnet, and you can never shrink it again. But Databricks documents only the new-subnet route for changing a workspace’s IP range, so that is the supported path, and it is not a quick edit: you stop every cluster and job, build the new subnet with its firewall rule and IAM grant, and point the workspace at it. Part 3 walks through the steps.

So size the subnet generously at the start. A /29 holds 8 addresses, and Google Cloud reserves 4 in every primary range, which leaves room for very few nodes; Databricks accepts anything from /29 to /9, and Part 3 shows how to size it from your peak number of nodes.

Databricks-managed or customer-managed VPC. Databricks has said it plans to deprecate classic workspace creation with a Databricks-managed VPC and recommends a customer-managed VPC for new workspaces. Treat customer-managed as the default.

What does Databricks create in your project?

This is the question a Google Cloud security team asks first, and the honest answer is short.

what workspace creation leaves behind: Databricks creates a workspace service account in its own project and grants it custom roles on your workspace project and on the subnet; in your project it enables two APIs, creates the databricks-compute service account and the workspace storage buckets, and later creates cluster VMs

Workspace storage buckets. In your workspace project Databricks creates databricks-<workspace-id>-system for system data (notebook revisions, job run details, command results and Spark logs) and databricks-<workspace-id> for the legacy DBFS root (the original Databricks file system, now a deprecated place for data). If Unity Catalog was enabled automatically, a third bucket backs the default workspace catalog. Never delete or modify these buckets: a workspace whose storage is deleted cannot be recovered. Do review their IAM members, and enable Data Access audit logs on them, as the Databricks documentation recommends. Part 5 covers them in depth.

Two service accounts and three custom roles. Workspace creation also leaves two Google identities behind. The workspace service account lives in Databricks’ own project, and the control plane uses it to start and stop cluster VMs in yours. databricks-compute lives in your workspace project, and every cluster node runs as it unless you attach a different one. Databricks grants the first one three custom roles in your projects. Who creates those roles, what each allows, why Databricks’ customer ID has to be on your allow list, and why none of it needs a key are identity questions, so Part 2 answers them in full. For this post, two facts are enough: the roles are ordinary IAM bindings in your project that you can read and revoke, and with a customer-managed VPC they include no permission to create or delete networks.

Before any of this, Acme’s network team builds the network in the host project acme-net-host and describes it to Databricks. In outline (not run here):

VPC     acme-dbx-vpc (custom mode)        Cloud Router  acme-dbx-router
subnet  dbx-prod-euw2, 10.10.0.0/22,      Cloud NAT     acme-dbx-nat
        Private Google Access on          firewall      allow traffic within the subnet
Databricks network configuration acme-prod-euw2-net -> workspace acme-prod-euw2-supply

Part 3 has the full gcloud commands, and the databricks_mws_networks and databricks_mws_workspaces Terraform that registers the network and creates the workspace.

What is a cluster made of on Compute Engine?

A Databricks cluster (also called classic compute) is a group of VMs that run Apache Spark together: one driver that coordinates the work and zero or more workers that do it. Databricks runs one Spark executor per worker. On Google Cloud each node is a plain Compute Engine VM in your subnet.

A simple example, back in the kitchen: a head chef reads the order and splits it into tasks, and line cooks each do a share. Each cook needs two phone lines, one to talk to the other cooks and one to call the office and the suppliers. The head chef is the driver, the line cooks are the workers, and the two phone lines are the two IP addresses every node takes from the subnet, as the diagram shows:

anatomy of a classic cluster on Compute Engine: one on-demand driver and Spot-capable workers, each a VM with two internal IP addresses, a 30 GB boot disk, a 150 GB container root volume and optional 375 GB local SSDs; the subnet size caps the number of concurrent nodes

Two IP addresses per node. One carries traffic inside the cluster and one carries traffic out to the control plane and data sources. So a subnet supports roughly half as many nodes as it has usable addresses: a /24 gives about 120 nodes and a /22 about 500. The subnet must be between /29 and /9, and its region must match the workspace region. Several workspaces can share one VPC, but each needs its own subnet. Acme gave each workspace a /22.
Three kinds of disk. A 30 GB boot disk for the host operating system and Databricks services, a 150 GB container root volume for the Spark runtime and logs, and optional local SSDs (disks physically attached to the host, 375 GB each) for Spark shuffle files and the disk cache. The disk cache keeps copies of remote data on local SSD and uses at most half of it.
Labels from tags. Databricks copies cluster and pool tags to labels (Google’s key-value metadata) on each VM and its disks, lowercased and cleaned to fit label rules. Up to 54 custom tags propagate, which is what makes per-team cost reports in Cloud Billing possible.

How does this apply to Acme? The nightly supply cluster has one head chef and between 2 and 16 line cooks: one n2-highmem-8 driver and up to 16 n2-highmem-8 workers. At full size that is 17 nodes and 34 addresses from the /22 subnet, well inside its room for about 500 nodes, and its cost_centre tag becomes a label on every one of those VMs and disks.

Which machine types should you use?

Compute Engine names machine types as family, shape, vCPUs. In n2-highmem-16, N2 is the family, highmem is the shape and 16 is the number of virtual CPUs. The shape tells you memory per vCPU: standard is typically 4 GB, highmem typically 8 GB and highcpu typically 2 GB (N2’s highcpu has 1 GB). Databricks’ compatibility reference for Google Cloud lists the families it uses, including E2, N1, N2, N2D, N4, N4D, C2, C3, C3D, C4, C4A and C4D for CPUs, and A2, A3, G2 and G4 for GPUs.

Two families deserve a note. C4A runs on Google’s Axion processor, an Arm CPU, and it is in Databricks’ list. Arm can give better price for performance, but check first that every library, JAR and init script you rely on has an Arm build. For GPUs, the Databricks GPU page lists A2 shapes (NVIDIA A100) and G2 shapes (NVIDIA L4); GPU compute needs a machine learning runtime and does not support Photon.

Spot VMs are spare Compute Engine capacity sold at up to a 91% discount, which Compute Engine can reclaim at any time with little warning. Google describes them as the latest version of preemptible VMs, and Databricks still uses the word preemptible in its settings. They suit workers in fault-tolerant jobs. Keep the driver on-demand, as Databricks advises: the driver coordinates the whole cluster, so losing it loses the work.

A simple example: a standby airline ticket is much cheaper, but you fly only if a seat is free and can be bumped. That is fine for passengers who can take the next flight, and never for the pilot. Workers are the standby passengers; the driver is the pilot. Acme’s nightly cluster, specified in the next section but one, follows exactly that rule: PREEMPTIBLE_WITH_FALLBACK_GCP puts workers on Spot capacity with a fallback to on-demand, and first_on_demand: 1 keeps the driver on a confirmed seat.

What happens when a cluster starts?

When something fails, it helps to know the order of events. This is a classic cluster launch, step by step, with the failure each step tends to produce:

what happens when a classic cluster starts, step by step: validation in the control plane, VM creation through the Compute Engine API in your project, network interfaces in your subnet, boot and disks, the outbound call to the secure cluster connectivity relay, Cloud Storage access through Private Google Access and library downloads through Cloud NAT or a mirror, and registration; each step has its own typical failure

Two failures deserve a mention because they are quiet: nothing goes wrong on the first day, and they appear only later, under particular conditions.

The subnet runs out of IP addresses. The network sizing guide says “each Databricks compute node requires two IP addresses from the subnet”, and what counts is the total number of nodes running at the same moment across every cluster in the workspace. A /26 subnet has 64 addresses, Google Cloud keeps 4, so 60 are usable: room for 30 nodes in total. One 10-node cluster starts happily on day one. Months later, when three teams each run a 12-node job at month end, the third cluster cannot get addresses and fails to start. Size for the peak, not the pilot.
A missing iam.serviceAccounts.actAs permission. Every cluster VM runs as a Google service account: databricks-compute by default, or a custom one attached with google_service_account (covered in the next section). Security teams often trim the workspace service account’s right to act as those accounts, and clusters then fail to start with a permission error, often weeks after the change. Part 2 shows which grant to restore.

Which Google-specific settings does a cluster have?

Every cluster definition on Google Cloud can carry a gcp_attributes block, the counterpart of aws_attributes and azure_attributes. Here is the job cluster for Acme’s nightly supply pipeline:

a job cluster spec for Acme's nightly supply pipeline with the Google-specific gcp_attributes block: preemptible workers with on-demand fallback, an on-demand driver, automatic zone choice, two local SSDs, a larger boot disk, and a tag that becomes a Compute Engine label

availability is ON_DEMAND_GCP (the default), PREEMPTIBLE_GCP or PREEMPTIBLE_WITH_FALLBACK_GCP. Fallback keeps a job running when spare capacity runs out.
first_on_demand places the first N nodes on on-demand VMs. Any value above zero keeps the driver on-demand.
zone_id is AUTO (Databricks picks one zone), HA (nodes spread across zones in the region, which can add inter-zone egress charges) or a named zone such as europe-west2-a, useful when you hold reservations there.
local_ssd_count and boot_disk_size size the disks. Local SSD can only be set on some machine families, such as N1, N2 and N2D; -lssd machine types come with a fixed number built in.
google_service_account attaches a custom service account to the nodes, which code on the cluster then uses to reach Cloud Storage and BigQuery. Acme leaves it out on purpose: every workload on that cluster would share that account’s Google permissions, with Unity Catalog not in the path. Part 2 explains the safer options.

What is an instance pool, and when do you need one?

Starting a classic cluster means asking Compute Engine for new VMs, booting them and loading the Databricks Runtime, which takes minutes. An instance pool shortens that wait. In Databricks’ words, pools “are a set of idle, ready-to-use instances”: VMs of one machine type that stay running, optionally with a Databricks Runtime version already loaded. A cluster that names the pool (the instance_pool_id field, or a pool chosen in the cluster form) takes its nodes from the pool instead of creating new VMs, and gives them back when it ends, ready for the next cluster.

You create a pool separately from clusters, under Compute, then Pools, or with the Instance Pools API or the Terraform resource databricks_instance_pool. Its main settings are:

Minimum idle instances: how many VMs it keeps running even when no cluster uses them. These “do not terminate” whatever the auto termination settings.
Maximum capacity: a ceiling on idle plus in-use instances.
Machine type (fixed once created), a preloaded runtime and tags.
Google Cloud settings in its gcp_attributes. One gotcha: in a pool the availability field is called gcp_availability, whereas in a cluster it is availability.

The cost rule is the reason to be careful: “Databricks does not charge DBUs while instances are idle in the pool. Instance provider billing does apply.” Idle VMs still appear on your Google Cloud bill, so the pool best practices suggest a minimum idle count of 0 unless start-up time really matters. For most new interactive work, serverless compute (later in this post) starts in seconds and makes pools unnecessary; pools remain useful for classic job clusters that must start quickly.

A simple example: a car hire desk that keeps a few cars warmed up at the kerb. Customers drive away at once, and the hire firm charges no rental fee while the cars wait, but somebody pays for the fuel they burn idling. The waiting cars are idle instances, the rental fee is DBUs, and the fuel is the VM billing on your Google Cloud bill. A quick test, cheapest answer first:

Does this workload need a pool?               Ask in order; stop at the first yes.
1. Can it run on serverless compute?          -> serverless, no pool
2. Are a few minutes of start-up acceptable?  -> a plain job cluster, no pool
3. Must a classic cluster start quickly?      -> a pool with minimum idle 0
4. Must it start at once, every time?         -> only then minimum idle above 0, paying for idle VMs

How does this apply to Acme? The nightly supply pipeline runs while nobody is waiting, so a few minutes of start-up cost nothing. It stops at question 2, and its job cluster names no pool.

How do instance types map from AWS and Azure?

If you are moving workloads from another cloud, map by shape, not by name:

mapping instance families from AWS and Azure to Google Cloud by workload shape: general purpose to n2-standard, memory optimised to n2-highmem, compute optimised to highcpu shapes, Graviton and Cobalt to the Axion-based C4A, and GPU families to G2 and A2; match vCPUs, memory and CPU architecture, then benchmark

My view: treat the table as a starting point, run your three heaviest jobs on the candidate shapes, and compare cost per job rather than cost per hour.

Classic or serverless?

Databricks recommends serverless compute as the primary option and classic compute only where serverless does not support your use case. On Google Cloud the deciding question is usually the one a security team asks:

classic versus serverless compute on Google Cloud: classic runs in your project under your network controls and is billed by Google; serverless runs in Databricks-managed projects, starts in seconds, is governed by account-level network settings and has a narrower feature set

Classic compute keeps the VMs in your project, so every Google Cloud control applies: VPC firewall rules, Cloud NAT, Private Google Access, Private Service Connect and VPC Service Controls. It supports R, custom containers, GPUs and the Spark UI. Serverless starts in seconds and removes cluster tuning, but it does not support R, its notebooks run Python and SQL only (Scala is not supported in notebooks, though JAR tasks in jobs are), it has no Spark UI, and it is governed by account-level network settings instead of your VPC.

Serverless does reach Cloud Storage and BigQuery privately. The serverless networking page (updated 29 October 2026) says that “traffic to Google APIs routes over the Google Cloud backbone”, so those calls never cross the public internet. To lock a bucket down to Databricks serverless only, you list the serverless project numbers in a VPC Service Controls perimeter (Private Preview as of October 2026). Separately, an account-level network connectivity configuration (NCC) can hold private endpoint rules, which create Private Service Connect connections from serverless to your own services behind a service attachment and an internal load balancer (Enterprise). One honest gap: a dedicated Private Service Connect endpoint from serverless to Google APIs themselves is only in Beta, on the page Configure private connectivity to Google APIs, so if your security policy insists on a PSC endpoint for Cloud Storage traffic, classic compute is still the generally available way to get one. Part 4 explains each path.

Back to the caterer one last time. Cooking in the caterer’s kitchen is quicker and needs nothing from you, so start there; cook at home only when the dish needs your own cooker, or when house rules say the food may not leave the building. As a test, in order:

Classic or serverless?
1. Does serverless support everything the workload uses?     no  -> classic
   (no R, no Scala notebooks, no Spark UI)
2. Does your security policy accept account-level network     no  -> classic
   settings, and Beta PSC to Google APIs, instead of your VPC?
3. Both yes?                                                       -> serverless; classic is only
                                                                      for what failed 1 or 2

How does this apply to Acme? Acme ends up with both. Its analysts use serverless SQL warehouses for dashboards, and the supply team’s nightly pipeline runs on classic job compute in acme-dbx-prod, because it uses Scala libraries and runs inside the generally available VPC Service Controls perimeter that Part 3 builds.

My view: start with serverless for SQL, BI and new pipelines, and keep a well-built classic workspace for the workloads that need your network or your libraries. Build the classic foundation properly even if you expect to use it rarely; it is the part you cannot change later.

What can go wrong?

Domain restricted sharing blocks creation. New organisations enforce it by default. Add Databricks’ customer ID C01p0oudw before the first attempt.
No organisation. The Databricks sign-up needs a project that belongs to an organisation. A personal project at the top of its own hierarchy does not qualify.
The wrong person can cancel the subscription. Cancelling deletes every workspace. Limit Billing Administrator and Organisation Owner on the billing account; Part 9 explains.
Picking a region before checking features. Serverless compute is in all 15 regions, but agent features, managed connectors and some AI functions are not. Check the regional table first.
CPU quota runs out per family. A cluster of N2 or C4A machines draws on its own regional pool. Raise quotas before go-live, not on the day.
The subnet is too small. Two IP addresses per node, counted across every cluster, pool and classic SQL warehouse that shares it.
Private Google Access is off. Nodes cannot reach Cloud Storage, so clusters fail while reading init scripts or writing logs. (Libraries from PyPI or Maven Central come over the internet through Cloud NAT, or from a mirror.)
Someone deletes a workspace bucket. The workspace cannot be recovered. Restrict IAM on the buckets and turn on Data Access audit logs.
A deny-all egress rule goes in too early. It also blocks PyPI and Maven Central, so library installs fail. Plan a mirror or an allowlist first.

What would I do now?

Ask the platform team for the effective organisation policies on the folder where Databricks will live, and check domain restricted sharing, resource locations and Shared VPC restrictions against the table above.
Plan few workspaces and fewer projects: one project per owner and environment that can hold several workspaces, one subnet per workspace, and a Shared VPC host project if your network team owns networking. Size each subnet for peak nodes times two.
Raise Compute Engine quotas in the target region for the machine families you plan to use, and wait for confirmation before creating workspaces.
Create the workspace with Terraform and a customer-managed VPC (Part 2 shows the keyless pipeline), then review IAM on the workspace buckets and enable Data Access audit logs on them.
Write two compute policies on day one: one for serverless-first SQL and notebooks, and one for classic job clusters with an on-demand driver, preemptible workers with fallback and mandatory cost tags.

What did we learn?

Databricks is a managed platform for building a lakehouse and everything on it; its control plane runs in Databricks-managed projects, the classic compute plane runs in your VPC using secure cluster connectivity, Cloud NAT and Private Google Access (or Private Service Connect on the Enterprise plan), and the serverless compute plane runs in Databricks’ account in your region.
A Databricks account holds identities, regional Unity Catalog metastores and workspaces; subscribing through Google Cloud Marketplace creates it, and one order links it to one Cloud Billing account, while VMs, disks and buckets bill to each workspace project.
A classic workspace lives in one project and one region for its whole life, but a project can hold several workspaces; with Unity Catalog, plan few workspaces, group them into projects by owner and environment, give each its own subnet, and attribute cost with tags.
Most failed first deployments are landing-zone problems: domain restricted sharing, resource locations, per-family CPU quotas and decisions such as the VPC that cannot be undone.
Workspace creation leaves workspace storage buckets and two service accounts (Part 2 covers who creates their roles and why no key is involved), and a cluster is a set of plain Compute Engine VMs with two IP addresses each.
Databricks recommends serverless first, and classic compute remains the choice when you need your own VPC controls, such as a generally available VPC Service Controls perimeter or a Private Service Connect endpoint for Google APIs, or libraries that serverless does not support; serverless still reaches Cloud Storage and BigQuery over Google’s backbone, not the public internet.

What comes next?

We now know where everything runs and what it creates in your project. The next question is who is allowed to do what, starting with the two service accounts and three roles that workspace creation left behind. In Databricks on Google Cloud, Part 2: Identity and access, IAM, service accounts, single sign-on and automation, we start from Google Cloud IAM first principles, go through every service account a workspace uses, compare a console deployment with keyless Terraform from CI, set up single sign-on and automation identities, and look at which ways of reading Google Cloud data from Databricks are keyless and which are not.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands is fictional.