Databricks on Google Cloud, Prerequisite 1: What a lakehouse is: storage, table formats and Spark

This post is about the ideas underneath Databricks, not Databricks itself. If you already know what a lakehouse is and how Delta Lake and Apache Iceberg work, you can skip it, and Prerequisite 2 on Google Cloud may be the better place to start.

This post starts from zero: what a lakehouse is, the open file formats that make one possible, and Apache Spark, the open-source engine that processes the files. Everything here is general knowledge that applies on any cloud and with any vendor. The Databricks platform itself, where each part runs and how you are billed, starts in Part 1: Foundations. Every later part of the series builds on these words.

The running example is the same fictional company as in the Production-ready agents series. Acme Consumer Brands keeps its marketing data in BigQuery and its supply data in Databricks on Google Cloud. In this post we look at the supply side: one table, supply_service, with one row per brand, region and week, showing units ordered and units delivered. This is what four of its rows look like:

supply_service: one row per brand, region and week, owned by supply, in Databricks; four of its 390 rows, for Acme Pure in Western Europe

Prerequisite 2 of the Production-ready agents series explains every column.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: what a lakehouse is and how a table format turns files into tables, in three minutes. The narration is a synthetic voice; the script and animation are mine.

What are the key ideas?

Six terms carry this whole post.

Data warehouse: a database built for analysis, where data is cleaned and loaded into tables before anyone queries it.
Data lake: a large store of raw files, in any format, kept cheaply in object storage (a service that stores files as objects in buckets, such as Cloud Storage). An object is one file’s bytes plus a little metadata, found by its name, and it is replaced whole, never edited in place.
Lakehouse: one copy of data kept as open files in object storage, with warehouse features such as transactions and fast SQL added on top.
Open table format: a published way of describing a set of files as one table, with a log of every change. Delta Lake and Apache Iceberg are the two you will meet.
Catalog: the service that knows which tables exist, gives each one a name and points to its current metadata, so that engines can find a table by name instead of by folder.
Compute engine: the software that reads the files and does the work. The best known one for lakehouses is Apache Spark, which spreads one job across many machines.

the key ideas as one stack: Parquet files in object storage such as Cloud Storage, an open table format such as Delta Lake or Apache Iceberg that turns the files into tables, a catalog that names the tables and points to their metadata, engines such as Apache Spark that compute on them, and the workloads on top; everything from the table format down is open files in your own bucket

Where do companies keep data for analysis?

A warehouse checks and tidies data before it stores it; a lake stores everything first and leaves the tidying for later. The lakehouse combines the two, so let us start with the two older ideas.

Picture two ways to run a library. In the first, a librarian checks every new book, catalogues it and shelves it in the right place before anyone may borrow it. Finding a book is quick, but nothing gets in until it has been processed, and odd items such as loose maps are awkward. In the second, every donation goes straight into a big storeroom, as it arrives, in whatever box it came in. Anything fits and storing it costs almost nothing, but each visitor has to dig, and two people can reshelve the same box at once. The first library is a data warehouse, the second a data lake, and many companies end up running both, with the same books in each.

A data warehouse is a database designed for questions over years of history. Data is loaded on a schedule, checked against a schema (the fixed list of columns and their types) as it arrives, and stored in the warehouse’s own managed storage. This is called schema on write. Warehouses are excellent at SQL and reporting, and less suited to raw files such as images and logs. BigQuery began as Google Cloud’s data warehouse, and Acme’s marketing team uses it that way. BigQuery now also works on lakehouse tables, as Does Google Cloud have a lakehouse? explains.

A data lake is the opposite design. You land every file as it is, in object storage, and decide how to read it later. This is schema on read. Storage is cheap and any tool can read the files. The problem is that plain files have no transactions: if a job fails halfway through writing, readers can see half a result, and there is nothing to stop two jobs from overwriting each other. On Google Cloud, a lake is usually a set of Cloud Storage buckets (Learning Cloud Part 4 explains them).

How does this apply to Acme? Acme has both libraries. Marketing’s campaign data sits in BigQuery, checked against a schema as it is loaded: the catalogued library. Supply’s delivery files from the distribution centres land in Cloud Storage as they arrive: the storeroom.

Many companies ended up running both, which a 2021 paper by four Databricks authors called the “two-tier data lake + warehouse architecture”. Data is loaded into the lake, then loaded again into the warehouse. The paper’s complaint is that this creates “complexity, delays, and new failure modes”: two copies, two sets of permissions, and reports that disagree.

What is an object?

An object is a sealed parcel: a label, contents the store never opens, and a tag of facts about it, replaced whole but never edited. Think of a left-luggage office. Each parcel has a label you use to collect it, the contents are sealed and the office never looks inside, and a tag records facts such as the weight and when it was handed in. You cannot slip one more item into a sealed parcel; you can only hand in a new parcel under the same label. And the shelf names the staff write on labels, such as “Shelf B”, are just part of the label: the room itself is one big floor.

Mapped to the real terms: the label is the object’s key, the sealed contents are its data, the tag is its metadata, and the shelf names are folder-like prefixes in a flat namespace.

Object storage deserves a closer look, because everything in a lakehouse sits on it. Cloud Storage keeps data in buckets (named containers that belong to a Google Cloud project), and a bucket holds objects. An object has three parts:

The key, or object name, such as bronze/deliveries/2026-05-04.json. It is unique within the bucket, and it is how you find the object.
The data: the file’s bytes. Cloud Storage does not look inside them. A JSON file and a Parquet file are both just bytes to it.
The metadata: name-value pairs about the object, such as Content-Type, the size, the generation number that Cloud Storage assigns, and any custom metadata you add, such as source = dc-rotterdam.

How does this apply to Acme? One delivery file from the Rotterdam distribution centre is one parcel: its label is bronze/deliveries/2026-05-04.json, its contents are the JSON bytes, and its tag carries Content-Type, the size, a generation number and source = dc-rotterdam.

Two details matter for everything that follows. First, the namespace is flat: there are no real folders. bronze/deliveries/ is just the start of the object’s name, and tools such as the Google Cloud console treat the slash as a separator so that it looks like a folder. (A bucket created with hierarchical namespace turned on does have real folders; this post assumes the flat kind.)

Second, objects are immutable. The Cloud Storage documentation says that “an uploaded object cannot change throughout its storage lifetime”. You cannot append a line to an object or change one value inside it. You can only upload a whole new object under the same name, which replaces the old one and gets a new generation number. If Object Versioning is turned on for the bucket, the old object is kept as a noncurrent version.

what an object is: a bucket holds objects in one flat list; each object is a key (its name), the data (its bytes) and metadata such as content type, size, generation and custom key/value pairs; folders are only name prefixes, and an object is replaced whole, never edited in place

That immutability is exactly why table formats need a log. If a table is many objects and none of them can be edited, then changing the table means writing new objects and recording, somewhere, which objects make up the table now. That record is the log you will meet in What is an open table format?.

Why are they called schema on write and schema on read?

A schema is the list of columns a table has and the type of each one, for example units_delivered as a whole number. Both designs use a schema. The names tell you when the schema is enforced.

Schema on write: the schema is checked when data is written into the table. A row that does not fit is rejected at load time, so everything inside the table already matches the schema.
Schema on read: files are written as they are, with no check. The schema is applied when someone reads them. Each reader decides how to interpret the columns, and bad values only surface at query time.

A memory aid that stays accurate: Write: Warehouse checks. Read: Reader decides.

The misconception to drop is that schema on read means “no schema”. Both have one; they differ in who enforces it, and when. An everyday version: a bank counter checks your paying-in slip while you stand there, and hands it back if the amount is unreadable. A night-deposit box accepts any envelope, and the problem is only found the next morning, by whoever opens it. The counter is schema on write; the deposit box is schema on read.

How does this apply to Acme?

Here is the difference on one Acme delivery file. The Rotterdam distribution centre sends a row where units_delivered arrives as the text "1,204", with a thousands separator, instead of the number 1204.

In a warehouse, the load checks every value against the column type. A BigQuery load job, for example, accepts no bad records by default: its maxBadRecords setting “is 0, which requires that all records are valid”. The load fails with an error, and no row is written until someone fixes the file.
In a lake, the file lands in Cloud Storage unchanged, and nothing complains until a reader queries it with a schema that says units_delivered is an integer. In Apache Spark’s default PERMISSIVE mode, the value it cannot parse becomes null, so the weekly total quietly comes out 1,204 units short. In FAILFAST mode, the query fails instead. A column that is missing from the file altogether also comes back as null.

schema on write versus schema on read, with one bad delivery row where units_delivered arrives as the text 1,204: a warehouse checks the row against the schema while loading and rejects the load; a lake stores the file as it is, and the problem only appears when a reader applies a schema at query time, as a null or an error

The lakehouse brings schema on write back, on top of the same files. In the Databricks documentation’s words, “Databricks validates data quality by enforcing schema on write for Delta Lake tables”: an insert whose columns do not exist in the table, or whose types cannot be safely cast to the table’s types, fails, and the table stays as it was. Changing the schema is still possible, but only on purpose. Schema evolution lets a write add new columns when you ask for it, with INSERT WITH SCHEMA EVOLUTION or the mergeSchema option, and you can always change the schema explicitly with ALTER TABLE ... ADD COLUMN.

What is a lakehouse?

The simple definition: a lakehouse keeps all the data once, as open files in cheap object storage, and adds the reliability and speed of a warehouse on top of those same files.

Back to the two libraries. Instead of copying every book from the storeroom into the catalogued library, you keep one storeroom and give it what the catalogued library had: a catalogue, a loan ledger that records every change, and rules about what may go on each shelf. Visitors from any library can use the same storeroom, because the catalogue and ledger are written in a published format. The storeroom is object storage, the catalogue is the catalog, the ledger and rules are the open table format, and the visitors are the engines.

A common misconception is that a lakehouse is one product you buy. It is a design, and several products implement it: Databricks is one, and Google Cloud’s Lakehouse, below, is another.

How does this apply to Acme? supply_service exists once, as files in Cloud Storage. Databricks reads and writes it, and, once Part 8 is in place, BigQuery reads the same files without a second copy.

The formal definition comes from that paper, Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics (CIDR 2021): a lakehouse is “a data management system based on low-cost and directly-accessible storage that also provides traditional analytical DBMS management and performance features such as ACID transactions, data versioning, auditing, indexing, caching, and query optimization.”

Broken into its parts:

Low-cost, directly accessible storage means object storage, where any engine that knows the format can read the files without going through one vendor’s server.
ACID transactions (atomic, consistent, isolated, durable) mean a write either fully happens or does not happen at all, and readers never see a half-written table. What does ACID mean? takes each letter in turn.
Data versioning and auditing mean you can see who changed a table, and read the table as it was at an earlier version.
Indexing, caching and query optimisation are the warehouse techniques for fast SQL, now applied to files. A section below explains each one.

Databricks also documents a common way to organise lakehouse tables, the medallion architecture: raw data lands in bronze tables, is cleaned and deduplicated into silver tables, and is shaped into business-ready gold tables. At Acme, delivery files from the distribution centres land in bronze, and supply_service is a gold table. Is the medallion architecture three copies of the data? answers the question this raises.

Does Google Cloud have a lakehouse?

Yes. Google Cloud’s lakehouse product is called Lakehouse. Until 20 April 2026 it was called BigLake. It keeps tables in the open Apache Iceberg table format, explained later in this post, as files in Cloud Storage. The Lakehouse runtime catalog is the serverless catalog that engines use to find those tables. BigQuery, Managed Service for Apache Spark, Apache Flink, Apache Hive and Trino can all work on the same Iceberg tables, so there is one copy of the data. BigQuery can write to these tables with ordinary SQL (DML and DDL) and looks after their maintenance automatically.

Cross-cloud data access goes a step further. Google first called it cross-cloud Lakehouse, and now uses the name borderless Lakehouse for the product as a whole. Cross-cloud data access has been generally available since 2 November 2026 (it was in Preview before). It lets BigQuery and Spark query Iceberg tables that stay in another cloud or another catalog, for example in Amazon S3 with an AWS Glue catalog, or in Databricks Unity Catalog, without copying the files to Google Cloud first.

So on Google Cloud, BigQuery and Databricks are both lakehouse engines, and both can read the same open files. The rest of this post explains the open ideas underneath, which apply to both.

a data warehouse, a data lake and a lakehouse compared: where data is stored, when the schema is applied, whether there are transactions, what each is good and weak at, and the Google Cloud example of each

What does ACID mean?

ACID means a change either lands completely and safely or not at all, and nobody sees it half done. The classic example is moving £100 from your savings to your current account:

Atomic: both halves happen, the £100 leaves savings and arrives in current, or neither does. You never lose £100 in between.
Consistent: the bank’s rules hold before and after, for example “a savings balance may not go below zero”.
Isolated: a statement printed during the transfer shows the balances before it or after it, never with the money in neither account.
Durable: once the app says “transfer complete”, a power cut a second later does not undo it.

A transaction is a group of changes that a system treats as one unit. ACID names the four guarantees a database gives about its transactions. The short version from the definition above still holds: a write either fully happens or does not happen at all, and readers never see a half-written table. Here is each letter, and how Delta Lake provides it on plain files in Cloud Storage. How does this apply to Acme? The same four letters, on supply_service:

Atomicity: all of a write’s changes happen, or none of them do. Suppose Acme’s nightly job must write 12 new Parquet files and fails after writing 7. Delta writes the data files first and commits last, and the commit is the single act of creating the next numbered JSON file in _delta_log. The failed job never created that file, so no reader sees any of the 7 files. The Databricks documentation puts it plainly: “Data files are not tracked unless the transaction log records a new version.” Later, VACUUM (a maintenance command) deletes such leftover files, together with old files the table no longer needs, once they are older than the retention period; What does VACUUM delete? explains exactly what goes.
Consistency: every commit moves the table from one valid state to another. In database theory this is partly the application’s job, because only the application knows all of its business rules. What Delta enforces is the rules you declare: schema enforcement (columns and types must match) and constraints, such as NOT NULL (a column may not be empty) and CHECK (a condition every row must meet, for example units_delivered >= 0). “When a constraint is violated, the transaction fails with an error,” and the table stays at its previous version.
Isolation: jobs running at the same time never see each other’s half-done work. Delta uses optimistic concurrency control: writers take no locks; each one does its work, then checks at commit time whether anyone else has committed a conflicting change in the meantime. If not, it commits. If so, it fails with a conflict error and can be retried. Readers always read a snapshot, one complete version of the table. By default, Databricks provides “snapshot isolation on reads and write-serializable isolation on writes”. The write level is called WriteSerializable: committed writes behave as if they had run one after another.
Durability: once a commit succeeds, it stays, even if every cluster crashes a second later. The commit file and the data files are objects in Cloud Storage, so the table has the same durability as the bucket it lives in.

the four ACID properties: what each one means, how Delta Lake provides it, and an example from Acme's supply_service table

How does Delta make the commit atomic? Version 2 of the table exists exactly when the object 00000000000000000002.json exists, so committing means creating that one object. The Delta protocol says that “Writers MUST never overwrite an existing log entry”, and that they should use “atomic primitives of the underlying filesystem to ensure concurrent writers do not overwrite each other’s entries”. Cloud Storage offers exactly this kind of primitive: an upload with a generation-match precondition of 0 “only proceeds if no object with the specified name exists”. This pattern is usually called put-if-absent. When two writers race for the same version number, only one can win:

optimistic concurrency in Delta Lake: two writers both read version 1 and both try to create 00000000000000000002.json; the storage lets only one create it, so writer A wins; writer B's create fails, B reads A's commit, finds no conflict because both only appended, and commits 00000000000000000003.json instead

What if both writers change the same rows?

The first writer to commit wins, and the second is told to start again on the new version; nothing is merged silently. You have seen this on a shared wiki page. Two people open the same page and both edit the same paragraph. The first to press Save succeeds. The second gets an “edit conflict” message: the page changed since they opened it, so they must look at the new version and redo their change on top of it. Nobody locked the page while they typed; the check happens only at save time. That is why it is called optimistic: it assumes conflicts are rare and checks at the end. The wiki page is the table, pressing Save is the commit, and the edit conflict is a conflict exception.

How does this apply to Acme? The race above ended well because both writers only appended. Now suppose writers A and B are two jobs that both run an UPDATE on the same Acme Pure row, and both have permission to do so. Permission is not what decides the outcome; optimistic concurrency is. Both read version 1, and both rewrite the Parquet file that holds the row, part-00002-e5f6.parquet. A commits version 2 first, and that commit removes part-00002. When B tries to commit, Delta checks what has changed since the version B read, finds that a file B read has been removed, and fails B’s write with a conflict exception. The documentation names each one by its cause:

ConcurrentDeleteReadException: “a concurrent operation deletes a file that your operation read”, as here.
ConcurrentDeleteDeleteException: “a concurrent operation deletes a file that your operation also deletes”, for example when both rewrite the same file.
ConcurrentAppendException: “a concurrent operation adds files in the same partition (or anywhere in an unpartitioned table) that your operation reads”.

B commits nothing, so A’s fix is never silently overwritten. B, or the job that runs it, must try again: read version 2, which already contains A’s change, re-run the UPDATE on top of it, and commit version 3.

a real write conflict: writers A and B both read version 1 and both update the same Acme Pure row, so both rewrite part-00002; A creates 00000000000000000002.json first; B's attempt finds that A removed a file B had read, so B fails with ConcurrentDeleteReadException and commits nothing; B's job retries on version 2, re-runs its update on A's new file and commits 00000000000000000003.json, so A's change is never silently overwritten

Whether two writes conflict depends on what each one reads and rewrites. Appends never conflict with each other, while two writes that rewrite the same data file do, as above, even when they change different rows of it. Databricks adds a feature called row-level concurrency that checks for conflicts row by row instead of file by file; Part 7 shows how Databricks implements this.

A failed write is simpler still. Data files that no commit names are not part of the table:

atomicity in Delta Lake: a job meant to write 12 Parquet files fails after writing 7; because it never creates the next commit file, the 7 files are orphans that no reader sees; readers keep using version 5, and VACUUM deletes the untracked files once they are older than the retention period

What do indexing, caching and query optimisation mean?

The lakehouse definition lists three more warehouse features. They serve one goal: answer a query while reading as little data as possible, as fast as possible.

A library makes all three easy to picture. Each shelf has a label such as “Authors A to C”, so you walk past every shelf whose range cannot hold the author you want: that is indexing, in the light form lakehouses use. The books you use every week sit on your desk instead of on a shelf at the far end: that is caching, as long as you notice when a newer edition arrives. And before you set off with a list of ten books, you plan the route so that you visit each aisle once: that is query optimisation. The real terms follow, then the same three ideas on an Acme query.

Indexing means keeping structures that let an engine find the rows it needs without reading everything. A classic database keeps a separate index, such as a B-tree, on chosen columns. A lakehouse uses lighter structures that suit files. Delta stores per-file statistics in its log (the minimum and maximum value, the null count and the row count for columns in each data file), and an engine can use them “at query time to skip irrelevant files”. This is called data skipping. It works best when similar values sit in the same files, and two classic layout techniques arrange that. Partitioning splits a table into separate folders by the value of a column (for example one folder per week_start), so a filter on that column reads only the matching folders. ZORDER rewrites files so that rows with similar values in several chosen columns end up in the same files, which makes the min/max ranges narrower and data skipping more effective.
Caching means keeping recently read data on fast storage close to the compute, so that the next read is quicker. Reading from object storage over the network is slower than reading a local disk, so an engine can keep a copy of the files it has read on the machine’s own disk or in memory, and must notice when the files change so that it never serves a stale copy.
Query optimisation means the engine choosing how to run a query: which files to read, in which order to join tables, and which join strategy to use. Spark SQL’s cost-based optimizer (CBO) uses table and column statistics “to estimate the cost of alternative plans and choose the plan with the lowest estimated cost”. Because estimates can be wrong, adaptive query execution (AQE) re-optimises the plan while the query runs, using the real size of the data produced so far. The engine then executes the plan.

data skipping: a query for week_start 2026-05-04 checks the minimum and maximum week_start that the Delta log stores for each data file, reads only the one file whose range can contain that date, and skips the other three; the optimiser plans the read, the engine executes it and a cache can keep a local copy for the next query

How does this apply to Acme? A query for week_start = 2026-05-04 on supply_service is the search for one author: the Delta log’s minimum and maximum week_start per file are the shelf labels, so the engine reads one file and skips the other three, as the diagram shows.

Databricks has its own versions of these ideas: a newer layout technique called liquid clustering that replaces partitioning and ZORDER, a disk cache on each worker’s local SSD, and a service that runs maintenance for you. Part 7 shows how Databricks implements each one.

Is the medallion architecture three copies of the data?

No: bronze, silver and gold are three increasingly refined forms of the data, not three identical copies. A restaurant kitchen works the same way. Deliveries go into the store room exactly as they arrived, in their crates, with the delivery note. Then the prep cooks wash, peel and portion them into labelled containers. Finally the line cooks turn those into the plated dishes that go to the tables. Nobody would call the crate of potatoes and the plate of chips “two copies”. If a prep step turns out to be wrong, you go back to the crates.

How does this apply to Acme? The crates are the bronze delivery files, the prepped containers are the silver delivery rows, and the plated dishes are the gold supply_service table. In the documentation’s terms:

They are three separate tables, each a more refined form of the data, produced from the layer before it. So yes, the data is physically stored more than once, but never as three identical copies. The Databricks documentation describes the goal as to “incrementally and progressively improve the structure and quality of data as it flows through each layer”.

Bronze keeps the raw data as received. It “is appended incrementally and grows over time”, has minimal validation, and often stores fields as strings so that nothing is lost when a source changes its format. It exists so that you can reprocess and audit: if a silver rule turns out to be wrong, you rebuild silver from bronze.
Silver is cleaned, typed, deduplicated and conformed (the same names and codes across sources). The documentation lists schema enforcement, handling of missing values, deduplication and type casting as silver work, and says silver “should always include at least one validated, non-aggregated representation of each record”.
Gold is shaped for the business: “often highly aggregated” and modelled for reporting, so it is much smaller. supply_service is a gold table.

one Acme delivery through the medallion layers: bronze keeps the raw JSON exactly as received, with the source file name, the ingest time and a duplicate from a re-sent file; silver holds one cleaned, typed and deduplicated row per delivery with a standard region name; gold holds one aggregated supply_service row per brand, region and week

How much is stored more than once depends on retention, which is your choice. Bronze holds every record as received, so it is usually the largest layer. Gold is so much smaller that its extra copy rarely matters. Delta also keeps the files of older table versions, for time travel, until VACUUM removes them (by default once they were removed more than 7 days ago), which adds storage in every layer. My view: decide on purpose how long bronze is kept, long enough to replay and audit, and write that decision down.

What is an open table format?

A table format is a ledger that says which files make up the table right now, and which made it up at every earlier version. Picture a library whose books are never altered once printed. To replace a page, it prints a new edition and records two lines in a numbered loan ledger: “page 41: removed edition 1, added edition 2”. To know what is on the shelf now, you read the ledger from the start, or from the last full stock-take. To know what was on the shelf last Tuesday, you stop reading at Tuesday’s page. The printed books are the Parquet data files, the ledger is the table format’s log, the stock-take is a checkpoint, and stopping early is time travel.

A lakehouse table is still a folder of files. The data itself is usually stored as Apache Parquet (a file format that stores data by column and compresses it well). What turns a folder of Parquet files into a table with transactions is a table format: a set of metadata files that records which data files belong to the table at each version.

Delta Lake is the format Databricks created and later open-sourced. Its documentation describes it as Parquet data files plus “a file-based transaction log for ACID transactions and scalable metadata handling”. The log, defined in the Delta transaction log protocol, is a folder called _delta_log next to the data. Every change to the table is a new numbered JSON file (00000000000000000000.json, 00000000000000000001.json and so on) listing the files added and removed. A reader that replays the log knows exactly which Parquet files make up the current table, and can also read any earlier version, which Delta calls time travel. Delta is the default table format on Databricks. For Acme, the ledger is the _delta_log folder next to the supply_service files; the sections below open it.

Apache Iceberg is another open table format with the same goal, defined by the Apache Iceberg table specification. It keeps its own metadata and snapshots (versions of the table), and an Iceberg catalog

points at the current metadata. Many engines can read and write Iceberg tables, BigQuery among them. On Google Cloud, Lakehouse manages Iceberg tables for BigQuery and the open-source engines.

Because both formats store data as Parquet and differ mainly in their metadata, Delta Lake offers UniForm (Universal Format). With UniForm turned on, the Delta writer also writes Iceberg metadata alongside the Delta log, so “a single copy of the data files supports both Delta and Iceberg clients”. At Acme, this is how marketing’s BigQuery queries will read supply’s Delta table without a copy; Part 8 shows how.

anatomy of a Delta table with UniForm turned on: Parquet data files, a _delta_log folder of numbered JSON commits that Delta readers such as Databricks use, and Iceberg metadata generated from the same commits for Iceberg readers such as BigQuery; both read one copy of the data files

How does it work? There is one copy of the Parquet files and two metadata layers. In the documentation’s words, “UniForm automatically generates Iceberg metadata asynchronously alongside the Delta Lake metadata, without rewriting the Parquet data files”. Turning it on is a matter of a few table properties, and it comes with conditions that follow from how Iceberg readers work.

The best known condition concerns deletion vectors. A deletion vector marks rows of a Parquet file as deleted in a small separate file, so a DELETE or UPDATE does not rewrite the whole Parquet file; readers apply the marks at query time. This approach is called merge-on-read, as opposed to copy-on-write, where the file is rewritten. An Iceberg v2 reader cannot apply Delta’s deletion vectors, so a table that Iceberg v2 readers must read through UniForm does without them, and every DELETE and UPDATE rewrites whole files. Iceberg v3 supports deletion vectors, which removes that trade-off for readers that support version 3 of the specification. Part 8 shows the exact table properties, the limits and how Databricks and BigQuery use them.

What does a Parquet file look like inside?

Parquet is a binary format: open a Parquet file in a text editor and you see unreadable bytes. It is designed to be read by programs. The Apache Parquet documentation describes the layout:

The file starts and ends with a 4-byte magic number, PAR1, which marks it as Parquet.
The rows are split into row groups. Inside each row group, the values of each column are stored together in a column chunk.
Each column chunk is split into pages, the unit that is encoded (for example with a dictionary, so a repeated value is stored once and then referred to by a small number) and then compressed (for example with Snappy or Zstandard).
At the end comes the footer, the file metadata: the schema, the number of rows, where each column chunk starts and, for each column chunk, statistics such as the minimum and maximum value. Then 4 bytes give the footer’s length, and PAR1 closes the file.

“File metadata is written after the data to allow for single pass writing.” A reader does the opposite: it reads the end of the file first, finds the footer, and fetches only the column chunks it needs.

the layout of a Parquet file, which is binary and not human readable: the magic bytes PAR1 at the start and end, row groups that each hold one column chunk per column, column chunks split into encoded and compressed pages, and a footer with the schema, row counts and min and max statistics for every column chunk, followed by the footer length

If Parquet were readable, the difference from a row format would look like this, using the four supply_service rows from the top of the post:

the four supply_service rows as they would read if Parquet were text: a row layout keeps each row together, a column layout keeps each column together; a query for the average otif_pct reads every value in the row layout but only one column in the column layout, and repeated values such as Acme Pure compress to almost nothing

What does a Delta log look like?

The Delta log is plain text that you can open and read. The protocol says that commit files “use newline-delimited JSON format, where every action is stored as a single-line JSON document”. An action is one change to the table’s state, such as adding a file. This is the first commit of supply_service, with each action wrapped onto several lines so that you can read it, and long values shortened:

the first commit of the supply_service Delta table, _delta_log/00000000000000000000.json: a plain-text file with one JSON action per line, commitInfo, protocol, metaData with the schema, and one add action per Parquet data file with its statistics; long values are shortened and each action is wrapped here for reading

Line by line:

commitInfo records who ran which operation, and when. The protocol lets engines store any JSON here.
protocol says which version of the Delta protocol a reader (minReaderVersion) and a writer (minWriterVersion) must support to use the table safely.
metaData gives the table a unique id, says the data files are Parquet ("provider":"parquet"), holds the schema as a JSON string in schemaString, and lists the partitionColumns (none here) and table settings (configuration).
Each add names one data file, with its path, size and modificationTime, and carries stats: the numRecords, minValues, maxValues and nullCount that data skipping uses.

The second commit shows why the log matters on immutable storage. Someone corrected one units_delivered value. Parquet files cannot be edited, so Delta wrote a new file containing the fix and recorded the swap as a remove of the old file and an add of the new one, in one atomic commit. (A table with deletion vectors can mark changed rows instead of rewriting the file; Acme’s table has UniForm turned on for Iceberg v2 readers, so it does without them.) The old file stays in the bucket, which is what makes time travel back to version 0 possible, until VACUUM deletes it after the retention period (see What does VACUUM delete? below).

Replaying thousands of small JSON files would be slow, so writers also write checkpoints from time to time: a Parquet file such as 00000000000000000010.checkpoint.parquet that holds the whole table state at that version. A small file called _last_checkpoint points to the latest one, so a reader starts there and replays only the commits after it.

the second commit, 00000000000000000001.json, records an UPDATE as a remove of the old Parquet file and an add of its rewritten replacement; further down, every so often a checkpoint file stores the whole table state in Parquet, and _last_checkpoint points to it, so a reader loads the checkpoint and replays only the commits after it

What does VACUUM delete?

Removed files do not disappear by themselves. A remove action only takes a file out of the current version; the file stays in the bucket so that older versions can still be read. VACUUM is the command that finally deletes files. The documentation describes it as removing “data files no longer referenced by a table that are older than the retention threshold”. In practice that covers two kinds of file:

files that earlier commits removed, such as part-00001-c3d4.parquet after the UPDATE above, once they were removed longer ago than the retention period;
files that no commit ever referenced, such as the leftovers of a failed write, once they are older than the retention period.

The retention period is the table property delta.deletedFileRetentionDuration, which is “interval 7 days” by default. After VACUUM, “the ability to query table versions older than the retention period is lost”, because their files are gone. VACUUM does not delete the log: “log files are deleted automatically and asynchronously after checkpoint operations and are not governed by VACUUM”, and how long they are kept is set by delta.logRetentionDuration, 30 days by default. Databricks can also run VACUUM for you; Part 7 shows how.

what VACUUM deletes, run on 21 May with the default 7-day retention: each bar is one data file, from the version that added it to the version that removed it; files removed before the retention line, and the untracked files of a failed job, are deleted; files still in the table or removed within the last 7 days are kept; the _delta_log files are not touched by VACUUM and expire separately after 30 days

What does Iceberg metadata look like?

Iceberg arranges its metadata as a tree. Only the top of the tree is text; the levels below it are binary.

The catalog stores, for each table, a pointer to its current metadata file. Committing a change means atomically swapping that pointer to a new metadata file.
The metadata file is plain JSON. The specification’s example name is v3.metadata.json; UniForm names its files <version-number>-<uuid>.metadata.json in the table’s metadata/ folder. It holds the format-version, a table-uuid, the table’s location, its schemas and partition-specs, the list of snapshots and the current-snapshot-id.
Each snapshot is one version of the table, and it names one manifest list.
The manifest list lists the manifest files for that snapshot, with counts of added, existing and deleted files. Each manifest lists data files, with each file’s path, format, row count and per-column lower and upper bounds, which play the same role as Delta’s minimum and maximum statistics. Manifest lists and manifests are stored in Apache Avro, a compact binary format.
The data files are Parquet, as before.

how an Iceberg reader finds the data for supply_service: the catalog points to the current metadata file, a readable JSON file listing schemas, partition specs and snapshots; the current snapshot names a manifest list, which lists manifest files, which list the Parquet data files with their statistics; manifest lists and manifests are Avro and data files are Parquet, both binary, shown here as if readable

This tree is the one used by versions 1 to 3 of the specification. With UniForm turned on, the Delta writer generates this Iceberg tree from the Delta log, and both point at the same Parquet files, as in the anatomy diagram earlier in this section.

How do Delta Lake and Iceberg compare?

Both formats do the same jobs on the same kind of files, in different ways. Side by side, using the Delta protocol and the Iceberg specification:

Delta Lake and Apache Iceberg compared job by job: how a reader finds the current state, where schemas and versions live, where per-file statistics are kept, how a commit is made atomic, which file formats the metadata uses, how row-level deletes work, and the data files, which are Parquet in both

Which should I use? The misconception is that you must pick one format for the whole company. You pick per table, by who writes it and who must read it. My rule of thumb, in the order I would check:

Question                                              Then
1. Is the table written and read only by Databricks?  Delta, the default. Stop here.
2. Must Iceberg-only readers, such as BigQuery,       Delta with UniForm: one copy,
   read a table that Databricks writes?               Iceberg metadata alongside.
3. Is the table written mainly by other engines,      Iceberg, managed where it is written.
   such as BigQuery or open-source Spark?
4. Only then: do readers need features that one       Check the limits in Part 8
   format's metadata cannot express?                  before choosing.

How does this apply to Acme? supply_service is written by Databricks and read by marketing’s BigQuery queries, so it sits at question 2: a Delta table with UniForm turned on, which is the design described above.

What is Apache Spark?

The Apache Spark project describes itself as “a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters”. In plain words, Spark splits one large job into many small tasks and runs them in parallel on a group of machines called a cluster.

An election count is a good picture. One returning officer plans the count and hands each counting table its own ballot boxes. Every table counts its boxes at the same time, and the officer adds the tables’ totals into one result. When the count is over, the counters go home, but the ballot boxes stay in the store, ready for a recount with more or fewer tables. The officer is the driver, the tables are executors on worker machines, the boxes are the data files, and the store is object storage.

A Spark cluster has two kinds of process. The driver runs your program: your notebook or job code. It plans the work and splits it into tasks. The executors run on the worker machines and carry out those tasks, each on its own slice of the data.

The key design choice is that storage and compute are separate: the data stays in object storage, so when the cluster shuts down the data is still there, and tomorrow you can start a bigger or smaller cluster.

How does this apply to Acme? When a job adds up units_delivered across Acme’s delivery files, the driver splits the files into tasks, each executor reads and sums its own slice of the Parquet files in Cloud Storage, and the driver combines the partial sums. When the cluster stops, the files and the supply_service table are still in the bucket.

Databricks builds on Spark with its own runtime and its own query engine, Photon; Part 7 explains both.

how Apache Spark runs a query: the driver turns your code into tasks and sends them to executors on worker nodes; each executor reads its own slice of the Parquet files from Cloud Storage in parallel, and the partial results come back to the driver

What can go wrong?

Treating a folder of Parquet files as a table. Without a table format there are no transactions: a job that fails halfway leaves half a result that readers can see, and two jobs can overwrite each other. Put a log on top, Delta Lake or Iceberg, before anyone relies on the data.
Expecting time travel beyond the retention period. After VACUUM, table versions older than delta.deletedFileRetentionDuration (7 days by default) can no longer be read. Raise it before you need an older version, and remember that it also keeps more files in storage.
Assuming concurrent updates simply merge. Two writers that rewrite the same file conflict, and the loser commits nothing. Jobs that update the same table at the same time need a retry, or a schedule that keeps them apart.
Finding bad data at query time. With schema on read, a value such as "1,204" quietly becomes null in Spark’s default PERMISSIVE mode. Enforce the schema and add constraints where the data enters the table, not in every report.
Keeping bronze forever by accident. Bronze is usually the largest layer, and old table versions add more until VACUUM runs. Decide how long each layer is kept, and write it down.

What would I do now?

Read the lakehouse paper once. It is short, and it explains why the two-tier lake plus warehouse design caused so much trouble.
Open the _delta_log folder of one real Delta table and read the first commit, line by line, against the Delta protocol.
Sketch your own medallion layers for one data source: what bronze keeps, what silver cleans and what gold aggregates, and how long each is retained.
List the engines that must read your most important table. If any of them speaks only Iceberg, note it now; Part 8 shows how UniForm serves both from one copy.
Go on to Prerequisite 2 for the Google Cloud side and Prerequisite 3 for identity and security, or straight to Part 1 if you already know both.

What did we learn?

A data lake stores files as objects in object storage: each object is a key, its bytes and some metadata, in a flat namespace, and it is replaced whole rather than edited, which is why table formats need a log.
Schema on write checks data as it enters a table and rejects bad rows at load time; schema on read stores files as they are and leaves each reader to interpret them at query time, so problems surface late.
A lakehouse keeps one copy of the data as open files and adds warehouse features on top: ACID transactions, schema enforcement, and fast SQL through data skipping, file layout, caching and query optimisation.
Delta Lake commits by creating the next numbered JSON file in _delta_log, which only one writer can do; a writer that loses a real conflict commits nothing and must redo its work on the newer version, and VACUUM later deletes files that no version inside the retention period (7 days by default) still needs.
Parquet data files are binary and columnar, the Delta log is plain JSON with Parquet checkpoints, and Iceberg metadata is a JSON file above Avro manifest lists and manifests; UniForm lets one copy of the Parquet files serve both, and the medallion layers are separate, more refined tables rather than identical copies.
Apache Spark splits one job into tasks that a driver hands to executors on a cluster, and because storage and compute are separate, the data outlives every cluster.

What comes next?

Prerequisite 2: What a cloud is, and what Google Cloud is covers the other half: Google Cloud’s regions, projects, billing accounts, IAM and the services this series uses, with a vocabulary map from AWS and Azure, and Prerequisite 3: Identity and security foundations covers single sign-on, OAuth, SCIM, keys and encryption. If you already know Google Cloud and those identity ideas well, go straight to Part 1: Foundations, how Databricks is built on Google Cloud, which starts with what Databricks is and builds on every idea in this post.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional.