Databricks on Google Cloud, from the basics

Databricks is often described as an AWS or Azure product, but it can also be deployed on Google Cloud, and has been since Google Cloud and Databricks announced their partnership in February 2021. As of October 2026, Databricks lists 15 Google Cloud regions, and its regional table shows serverless compute available in all of them.

In my work with partners, the projects that run Databricks on Google Cloud usually bring two groups of people into one room. One group knows Databricks well, often from AWS or Azure, and is meeting Google Cloud projects, service accounts and a global VPC for the first time. The other group knows Google Cloud well and has never had to think about a lakehouse, a DBU or a Unity Catalog metastore. Both groups are experts, and both are missing half of the picture.

This series fills in both halves, from the basics up to the level of detail an architect needs: real resource names, IAM roles, network ranges, API names, Terraform resources and gcloud commands, with the limits and failure modes that go with them.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: what the Databricks on Google Cloud series covers, and where to start, in three minutes. The narration is a synthetic voice; the script and animation are mine.

Who is this series for?

Databricks engineers and architects who are new to Google Cloud. You know clusters, Unity Catalog and Lakeflow. You want to know what a Google Cloud project is, how IAM and service accounts work, and what Databricks creates in your project.
Google Cloud engineers and architects who are new to Databricks. You know projects, VPCs and IAM. You want to know what a lakehouse is, how Databricks is built, and how it uses the Google Cloud services you already run.
Platform, security and network teams who have to approve, build or run a Databricks deployment on Google Cloud and need a shared vocabulary with the data team.

two ways into the series: if you know Databricks but not Google Cloud, start with Prerequisite 2; if you know Google Cloud but not the lakehouse, start with Prerequisite 1; both paths then read Prerequisite 3 on identity and security and meet at Part 1, which introduces Databricks itself, and continue through Part 10

Why does Databricks on Google Cloud need its own series?

The Databricks platform is the same on every cloud; the cloud underneath it is not. Your notebooks and tables look identical, but the machines, network, identities and bill they sit on are Google Cloud’s, and that is the half most teams have not met before.

A simple way to picture it is a coffee chain opening its first branch in a new country. The menu, the recipes and the till software are the same as at home. The building, the electricity supply, the staff badges, the local bank and even some of the words are new: ask for “chips” in London and in New York and you get two different things. Nobody would hand the new branch manager the old country’s lease and wiring plans and expect them to work.

Mapped to Databricks: the menu and recipes are the Databricks platform itself, the building and wiring are the cloud’s virtual machines and network, the staff badges are workload identities, the local bank is how you pay, and the confusing words are terms such as “account” and “workspace”.

Now the real terms. The Databricks platform is the same on every cloud. Its documentation describes the same split on AWS, Azure and Google Cloud: a control plane that Databricks runs in its own account, and a compute plane where your data is processed, either in your own cloud account or in Databricks’ serverless environment.

What changes is everything underneath. On Google Cloud, a workspace’s classic compute runs as Compute Engine VMs inside a VPC in your project. Identities for workloads are Google service accounts. Private connectivity uses Private Service Connect. You buy Databricks through Google Cloud Marketplace, and the charges appear on your Google Cloud bill. Words such as “project”, “account”, “workspace” and “service principal” mean different things on each side, which is where most of the confusion I see comes from. Each of those topics has its own part.

Where people get tripped up is assuming that, because the Databricks screens look the same, the platform checklist from AWS or Azure carries over unchanged. It does not: the recipes travel, the building has to be planned again.

How does this apply to Acme? Acme’s supply team works with the same Databricks notebooks, Lakeflow pipelines and Unity Catalog tables as a team on any other cloud. Underneath, its workspace’s classic compute runs as Compute Engine VMs in a VPC in Acme’s own Google Cloud project, its workloads act as Google service accounts, and the Databricks subscription appears on Acme’s Google Cloud bill. One word alone shows the trap: Acme has a Databricks account, a Cloud Billing account and several service accounts, and they are three different things. Prerequisite 2 lists the words that mean different things on each side.

What is the running example?

One made-up company, with two halves of its data, appears in every post, so each new idea lands on tables and teams you have already seen instead of a fresh toy example every time. It works like a cookery course that cooks the same dinner party all term: by the end you know every dish, so you notice what each new technique changes.

The same fictional company runs through this series and the Production-ready agents series. Acme Consumer Brands sells household products. Its marketing team keeps campaign data in BigQuery (Google Cloud’s data warehouse). Its supply team keeps delivery data in Databricks (a data and AI platform built on the lakehouse), running on Google Cloud. In the agents series, the two halves of that data are put side by side to answer whether a campaign worked; the business and data prerequisite explains the tables. Here is a quick look at both, three weeks of each:

a quick look at the two halves of Acme's data: campaign_performance has one row per campaign, brand, region and week, owned by marketing in BigQuery; supply_service has one row per brand, region and week, owned by supply in Databricks; three real rows of each

The business and data prerequisite explains every column and how the two tables join.

In this series, we build and run the Databricks half properly: the subscription and the projects, identity, networking, storage, governance, the link to BigQuery and Gemini, cost and operations. Acme and its data are made up; the products, settings and limits are real.

the running example: Acme Consumer Brands runs Databricks on Google Cloud for supply data and BigQuery for marketing data, side by side in Google Cloud; both read open tables in Cloud Storage, Unity Catalog governs the Databricks side, and Gemini Enterprise is where Acme's people ask questions

What are the three prerequisites and ten parts?

Three prerequisites give you the words; ten parts build, secure, connect, run and move the platform, in that order. Think of building and living in a house: first learn the materials, then build the house (Parts 1 to 5), then fit the locks and alarms (Parts 6 and 7), and finally connect it to the street, pay the bills and move your belongings in (Parts 8 to 10). For Acme, that means its Databricks estate in europe-west2 grows from empty Google Cloud projects into a governed platform that works alongside BigQuery and Gemini.

the thirteen posts in four stages: three prerequisites for the basics; Parts 1 to 5 build the platform; Parts 6 and 7 run it safely; Parts 8 to 10 connect it to Google Cloud, operate it and migrate to it

The basics

Prerequisite 1: What a lakehouse is: storage, table formats and Spark. Object storage, data warehouses, data lakes and the lakehouse; schema on write and schema on read; ACID transactions; indexing, caching and query optimisation; the medallion pattern; Parquet, the Delta Lake log, VACUUM, Apache Iceberg and UniForm; and Apache Spark. It covers the open ideas only: Databricks itself starts in Part 1.

Prerequisite 2: What a cloud is, and what Google Cloud is. Google Cloud’s regions and zones, the resource hierarchy, billing accounts and Marketplace, IAM and service accounts, the Google Cloud services the series uses, and a vocabulary map from AWS and Azure, including the words that mean different things.

Prerequisite 3: Identity and security foundations. Identity, authentication and authorisation; identity providers and single sign-on; SAML, OAuth 2.0 and OpenID Connect; ID tokens, access tokens and JWTs; the OAuth flows; SCIM; the many meanings of federation; machine identities and why keys are risky; MFA, RBAC and ABAC; encryption in transit and at rest; and IP ranges, CIDR notation and DNS. It covers the general ideas only: the Databricks on Google Cloud specifics are in Part 2 and later parts.

Build the platform

Part 1: Foundations, how Databricks is built on Google Cloud. What Databricks is and its main products; the control plane and the two compute planes; accounts, workspaces and metastores; Private Service Connect basics; subscribing through Google Cloud Marketplace and who pays for what; how projects map to workspaces, and where a workspace lands on Google Cloud, AWS and Azure; the organisation policies to check first; what Databricks creates in your project; clusters on Compute Engine; and choosing classic or serverless compute.

Part 2: Identity and access, IAM, service accounts, single sign-on and automation. Google Cloud IAM from first principles, keyless access, the service accounts a workspace uses, two deployment paths side by side (the console, and keyless Terraform from CI), user sign-in and provisioning, where each kind of role lives, which federation to use when, automation identities, and access to Cloud Storage and BigQuery from Databricks.

Part 3: Networking: VPC, Cloud NAT, Private Google Access and Private Service Connect. The customer-managed VPC and Shared VPC, subnet sizing, firewall rules, Cloud NAT, Private Google Access, how the control plane reaches clusters that have no public IP, back-end and front-end Private Service Connect and the DNS behind it, and VPC Service Controls.

Part 4: Serverless networking and egress. How serverless compute reaches Cloud Storage, BigQuery, your private services and the internet, network connectivity configurations, network policies that lock egress down, VPC Service Controls, and what serverless networking costs.

Part 5: Storage, Cloud Storage and how Unity Catalog uses it. Cloud Storage for a lakehouse, the bucket settings that matter for Delta tables, why hierarchical namespace buckets cannot be external locations, workspace storage and default storage, storage credentials and external locations, managed and external tables and volumes, and what DROP really deletes.

Run it safely

Part 6: Governance and security. Unity Catalog privileges and ownership, row filters, column masks and tag-based policies, lineage, customer-managed encryption keys in Cloud KMS, where secrets should live, and audit logs.

Part 7: The platform on Google Cloud, runtime, Photon, SQL, Lakeflow and AI. The Databricks products, from Part 1, configured and used on Google Cloud: the Databricks Runtime and Photon, what makes queries fast, SQL warehouses, Lakeflow, dashboards, Genie and Databricks Apps, Lakebase, agents, and which of it Acme can use in europe-west2.

Use it and move to it

Part 8: Better together, BigQuery, Gemini and Google Cloud services. The four ways Databricks and BigQuery read each other’s tables, Iceberg reads (UniForm) on a Delta table, Gemini models inside and outside Databricks, streaming from Pub/Sub, Looker, OpenSharing, and a reference architecture for Acme.

Part 9: Commercials, quotas and operations. Premium or Enterprise, DBUs and the one Google Cloud bill, Marketplace orders and private offers, who can cancel the subscription, who supports what, Google Cloud quotas and Databricks limits, cost levers, billing system tables and day-two operations.

Part 10: Migration to Google Cloud. Moving Databricks workloads from AWS, Azure or Hadoop to Google Cloud: what is different, what moves easily and what does not, a five-step approach, three routes for the data, a translation cheat sheet, reference architectures, and a recap of the whole series.

How should I read it?

Read the prerequisite you need, then the parts in order. Prerequisite 1 is for anyone new to the lakehouse, and Prerequisite 2 for anyone new to Google Cloud. Prerequisite 3 is for everyone who has not worked with SAML, OAuth, SCIM and encryption keys before; Part 2 and Part 6 link back to it rather than repeating it. Databricks itself is introduced in Part 1, so even readers who skip all three prerequisites should start there. Each part assumes the vocabulary of the ones before it.

If you are not sure where you fit, use this guide. It starts with the shortest path:

If you...                                      Read
know Google Cloud, Databricks and SSO well     Part 1, then the parts in order
need one topic only (say, networking)          Part 1, then that part and the links it gives
know Databricks, new to Google Cloud           Prerequisite 2, then 3, then Part 1
know Google Cloud, new to the lakehouse        Prerequisite 1, then 3, then Part 1
are new to both                                Prerequisites 1, 2 and 3, then Part 1

The common mistake is to skip straight to the part that looks urgent, such as networking, without Part 1. Most of the later parts lean on its picture of the control plane and the compute planes.

Three conventions run through the series:

Dates and stages. Databricks and Google Cloud both ship often. I date status claims “as of October 2026”, and label anything that is in Preview or Beta (available to try, but not yet generally available).
Commands and code. Every gcloud command, Terraform resource and configuration snippet is checked against the public documentation. Where I have not run one myself, I say so.
Public sources only. Every fact links to public documentation in the Sources section of its post. Opinions are labelled as mine.

What do I need to follow along?

Nothing, if you only want to read. If cloud computing itself is new to you, my Learning Cloud series covers the basics first. If you want to build as you go, you will need a Google Cloud organisation and project, and a billing account on which you hold the Billing Administrator role (roles/billing.admin), because that is what the Databricks subscription documentation requires to subscribe through Google Cloud Marketplace. Part 1 walks through the rest.

What did we learn?

Databricks has run on Google Cloud since the partnership announced in February 2021, and as of October 2026 it is available in 15 Google Cloud regions.
The Databricks platform is the same on every cloud; what changes on Google Cloud is everything underneath it, from projects and service accounts to Private Service Connect and Marketplace billing.
The series is written for two groups at once: Databricks people who are new to Google Cloud, and Google Cloud people who are new to Databricks.
One fictional company, Acme Consumer Brands, runs through every post, with marketing data in BigQuery and supply data in Databricks.
Three prerequisites cover the basics (the lakehouse, Google Cloud, and identity and security), and ten parts build, secure, connect, operate and migrate the platform.

What comes next?

Start with Prerequisite 1: What a lakehouse is: storage, table formats and Spark if the lakehouse is new to you, or Prerequisite 2: What a cloud is, and what Google Cloud is if Google Cloud is new to you. Then read Prerequisite 3: Identity and security foundations if SAML, OAuth, SCIM or customer-managed keys are new to you. All paths meet at Part 1: Foundations, how Databricks is built on Google Cloud, where Databricks itself is introduced and the two halves come together.

Sources

Opinions are my own. Every fact in this series comes from public documentation, linked in each post. Acme Consumer Brands and its data are fictional.