Iceberg Won the Format War. The Catalogue Is the Next One.
For peopleBy glitchdata Team 5 min read
💡 Key Takeaways
- Iceberg v3 reached general availability on the major platforms during the first half of 2026, bringing deletion vectors, row-level lineage and a variant type for semi-structured data.
- Deletion vectors and the variant type are designed to work across both Delta and Iceberg without rewriting data, which is what has taken the heat out of the format argument.
- Every Iceberg table has exactly one authoritative record of which metadata file is current — and whatever holds that pointer is where governance, audit and the real lock-in now live.
For about five years, choosing a table format was a decision people wrote strategy documents about. Delta or Iceberg, and by implication Databricks or everyone else. That argument is substantially over, and it ended in a way few of the strategy documents predicted: not with one format winning on merits, but with the differences being engineered away.
What Iceberg v3 Actually Added
Iceberg v3 reached general availability across the major platforms during the first half of 2026 — Snowflake, Databricks and Amazon S3 Tables among them. Three features matter.
Deletion vectors. Rows can be deleted or updated without rewriting the Parquet files that contain them. Databricks' own figure is up to a tenfold speedup on merge-heavy workloads against the previous copy-on-write behaviour, which is the difference between a nightly job and a continuous one for a lot of change-data pipelines.
Row-level lineage. Every row carries a unique identifier, and it is required for all v3 tables rather than optional. The practical effect is change data capture as a property of the table rather than as a separate piece of infrastructure bolted alongside it — which removes one of the more reliably fragile components of a modern warehouse.
The variant type. Semi-structured data — JSON and its relatives — stored natively in a column, with field extraction available through shredding rather than requiring the whole document to be parsed on read. Anyone who has stored event payloads as a text column and regretted it knows the shape of the problem.
Databricks states that the variant type and these features work across both Delta and Iceberg without rewriting data. That claim, more than any individual feature, is what deflated the argument. If the capability you were choosing a format for is available in both, the format is no longer the choice.
So What Is The Choice Now
Here is the thing about an Iceberg table that is easy to miss until it matters.
The table is a pile of Parquet files and metadata files in object storage. Working out which state is current means knowing which metadata file is the current one, and that pointer lives in exactly one place: the catalogue. Polaris, Unity Catalog, AWS Glue, Nessie, or something a vendor runs for you.
That single pointer is the entire lakehouse. Lose access to the catalogue and you do not have tables, you have a bucket full of Parquet and a reverse-engineering project. Control of the catalogue is control of who can read, who can write, what is audited and what is governed.
Which means the lock-in moved. It used to live in the format, and it was visible: a proprietary format was obviously a commitment. It now lives in the metadata service, and it is much less visible, because everything underneath it is genuinely open and the marketing is accordingly enthusiastic about openness.
Apache Polaris became a top-level Apache project in February 2026 and implements the Iceberg REST specification in full, which makes it the lowest-lock-in option and also the one you have to operate. Unity Catalog is open-sourced but governed by Databricks. Glue is AWS. Each is a reasonable choice; none of them is neutral, and the distinction is not the one the word "open" is usually asked to carry.
The Question To Ask
"Is our data in an open format?" was the right question in 2021 and is close to meaningless now, because the answer is almost always yes and it almost always was going to be.
The question for 2027 is narrower and ruder: if we stopped paying, who would still be able to tell us which metadata file is current? If the answer is a service only one vendor can run, then the openness of the format underneath is a property you own and cannot currently exercise. That is not necessarily a mistake — plenty of people buy that trade deliberately and get good value — but it is worth buying on purpose.
Our guides on data warehouses, data lakes and lakehouses, data warehouse vs lakehouse and building a data catalogue cover the groundwork, and the Databricks, Snowflake and DuckDB courses take the platforms one at a time. The DuckDB one exists partly to make the preceding question unavoidable: a great many lakehouses are holding data that fits comfortably on one machine.
Why We Care About This Here
A short note on our own position, since it is adjacent. Everything in the hub is a link rather than a copy: a dataset here points at the file where its publisher hosts it, and we store no copy of it. That is the same instinct as the question above, applied to a catalogue of other people's work. The index is useful precisely because it is not the only place the data exists.
Where to check this yourself: Databricks' own write-up of Iceberg v3 on the lakehouse is the clearest vendor account, and the Apache Iceberg specification is the thing none of the vendors can quietly change.