Lakehouse Table Formats in 2026: Iceberg, Delta Lake, Hudi, Paimon, and DuckLake, How They Work, Where They Stand, and Where They're Going

First Principles: What a Table Format Actually Does

One compressed refresher, because every comparison below builds on it. Object storage gives you files. Analytics needs tables: things with schemas, transactions, history, and the ability to change safely while being read concurrently. A table format is the metadata layer that closes that gap.

Every format on this list, whatever its architecture, must answer the same five questions:

• Which files constitute the table right now, without listing storage?
• How does a change, however many files it touches, become visible atomically?
• How do row-level updates and deletes work on immutable files?
• How do schemas and partitioning evolve without rewriting history?
• And how do engines plan queries efficiently, pruning most data before reading any?

The formats differ in their answers, and the differences trace back to birthplaces. Hudi was born at Uber in 2016 to make incremental ingestion and upserts possible on Hadoop-scale lakes. Delta was born at Databricks in 2017 to give Spark reliable transactions and unified batch-and-streaming. Iceberg was born at Netflix in the same era to fix correctness and scale for enormous multi-engine estates, with the spec-first neutrality that later won the standards war. Paimon was born in the Flink community at Alibaba in 2022 to make the table itself behave like a streaming system. And DuckLake was born in the DuckDB world in 2025 to ask whether the whole metadata apparatus had become more complicated than the problem.

Five origin stories, five design centers, and the rest of this article walks them one at a time.

Apache Iceberg: The Standard, Mid-Transformation

How it works. Iceberg's architecture is the snapshot tree my series has dissected at length: data files in Parquet, manifests listing those files with per-column statistics, manifest lists collecting manifests into snapshots, and a top-level metadata file whose pointer, swapped atomically through a catalog, defines the current table. Planning reads metadata, never lists storage, and prunes aggressively on statistics. Row-level change runs copy-on-write or merge-on-read, with v3's deletion vectors, compressed bitmaps in Puffin files, one per data file, making the merge-on-read path fast and stable. Partitioning is hidden, defined as transforms on columns and free to evolve, and schema evolution is by field ID, safe by construction. Every snapshot persists until expired, giving time travel and rollback natively.

Current state. The v3 spec, ratified mid-2025, is the production frontier, and the 1.10 and 1.11 release line through May 2026 delivered it: deletion vectors, the variant type with shredding for semi-structured data, row lineage for change tracking, and geospatial types. Ecosystem position is, frankly, the story: beyond the engines that always spoke it - Spark, Flink, Trino, Dremio - the last holdouts converged, with Databricks supporting Iceberg through UniForm and managed Iceberg in Unity Catalog, DuckDB shipping full read-and-write including UPDATE and DELETE, Snowflake open-sourcing pg_lake to bridge PostgreSQL directly to Iceberg tables, and S3 Tables making Iceberg a managed storage primitive with cross-region replication. The catalog ecosystem, anchored by the REST specification and implementations like Apache Polaris, is where governance consolidated, the subject of my Polaris article.

Roadmap. The v4 design cycle, which I covered thread by thread in my v4 state article, is the most consequential format work in the industry right now: single-file commits and an adaptive metadata tree to make streaming-rate writes cheap, snapshot offloading and columnar metadata to cut planning costs, relative paths and a typed content-stats model already ratified by vote, and the contested efficient-column-updates proposal aimed at AI-scale wide tables. Timeline honesty from that article stands: direction locked, dates not.

Pros. Broadest multi-engine read-and-write support in existence, neutral Apache governance no vendor controls, the deepest catalog and governance ecosystem, spec-first discipline that has kept implementations honest, and the network effect of being the format everyone else builds bridges to.

Cons. The metadata tree that scales so well is genuinely complex to operate and reason about, and heavy small-commit workloads expose its write amplification - the very thing v4 exists to fix. Streaming CDC is well supported but not the native idiom it is in Paimon or Hudi, with equality deletes as the pragmatic and read-costly bridge. And table maintenance - compaction, expiration - is your responsibility or your platform's, never free.

Delta Lake: The Ecosystem Heavyweight at a Crossroads

How it works. Delta's design center is the transaction log: a _delta_log directory of ordered JSON commit files, each recording actions - add this file, remove that one, change the schema - with periodic Parquet checkpoints compacting the log for fast reads. The current table is the log replayed to its tip, and commits are atomic appends of the next numbered log entry. It is a simpler mental model than Iceberg's tree, sequential rather than hierarchical, and it shaped a decade of Spark-centric lakehouse practice. Row-level change matured along the same arc as Iceberg's: copy-on-write first, then deletion vectors for merge-on-read. Layout optimization evolved past static partitioning into liquid clustering, incremental automatic clustering on chosen keys. Row tracking gives row-level lineage for change data feeds. And UniForm generates Iceberg-compatible metadata alongside Delta's own, letting Iceberg readers consume Delta tables - the bridge that acknowledged where interoperability gravity had settled.

Current state. Delta 4.0 arrived with Spark 4.0, and the 4.0.x line through 2026 tells you exactly where the project's center of mass sits: the headline features are catalog-managed tables, a commit model coordinated through the catalog rather than the storage-level log alone, and Unity Catalog OAuth integration, with UniForm's Iceberg interoperability restored on the new Spark line. Delta Kernel, the library that lets connectors implement Delta once against a stable core, continued maturing as the answer to the ecosystem's historical Spark-centricity, and delta-rs serves the Python and Rust world. The honest ecosystem read: inside the Databricks orbit, Delta is the best-integrated table format on earth, and outside it, engine support is real but consistently a tier shallower than Iceberg's, with Unity Catalog as the governance center of gravity rather than a neutral catalog layer.

Roadmap. The fascinating one. As I reported in my Iceberg v4 article, Databricks has proposed that Delta Lake 5.0 adopt the Iceberg v4 metadata tree as its native content metadata - one on-disk structure readable and writable by both formats' clients with no translation layer. Read that plainly: the largest Delta stakeholder proposing that Delta's next major version converge onto Iceberg's next metadata design. It rides on the v4 work rather than driving it, its fate belongs to the Delta community, and whether it lands fully or partially, it marks the endpoint of the format war more decisively than any market-share chart: the remaining differences are being engineered away at the metadata layer itself.

Pros. Unmatched integration depth in the Spark and Databricks world, the simplest core mental model of the mature formats, genuinely excellent engineering in deletion vectors, liquid clustering, and change data feed, and a convergence path that protects existing investments.

Cons. Gravity: the format is open, and its momentum, governance reality, and best features orbit one vendor, with Unity Catalog as the governance answer in a way that neutral-catalog shops find binding. Multi-engine write support outside that orbit trails Iceberg's. And the 5.0 convergence question, until resolved, puts a strategic asterisk on long-horizon Delta-native investments.

Apache Hudi: The Database Ambition

How it works. Hudi's design center has always been the most database-like of the five: the table as a timeline of actions - commits, compactions, cleanings, rollbacks - with data organized into file groups that function like micro-partitions of a storage engine. Copy-on-write and merge-on-read were Hudi concepts before they were industry vocabulary, with merge-on-read pairing base Parquet files with row-oriented log files that compaction folds in. The differentiator is indexing: Hudi maintains a multi-modal index subsystem in an internal metadata table, including a record-level index mapping keys to file groups, which makes keyed upserts fast in a way pure scan-and-match designs cannot be, plus bloom and secondary indexes to accelerate lookups beyond the key.

Current state. Hudi 1.0 reached general availability as the project's largest re-architecture: an LSM-structured timeline, non-blocking concurrency control designed so that writers, compaction, and clustering proceed without locking each other out, secondary indexes as a first-class subsystem, and functional indexes extending pruning beyond partitions. The 1.x line through 2026 has been hardening that foundation. Ecosystem-wise, Hudi remains strongest where it was born - Spark and Flink ingestion pipelines - with commercial energy concentrated in Onehouse, founded by Hudi's creator, whose interoperability stance is notable: Onehouse championed Apache XTable, the incubating project that translates metadata among Iceberg, Delta, and Hudi, effectively conceding that consumption happens in other formats' ecosystems while competing on ingestion and table management excellence.

Roadmap. The 1.x arc points at deepening the database analogy: richer indexing, faster keyed operations, tighter streaming semantics, and positioning Hudi as the write-optimized ingestion tier of a multi-format world, with XTable and Iceberg-compatible surfaces handling the read-side reach.

Pros. The strongest keyed-upsert machinery of the file-metadata formats, genuine innovations in non-blocking concurrency, the most complete built-in table-services story - ingestion, clustering, cleaning as managed subsystems - and years of production hardening at extreme CDC scale.

Cons. Operational complexity is the recurring field report: more concepts, more configuration surface, more tuning than its peers - a real cost for teams without dedicated platform depth. Multi-engine read support, while broad on paper, is shallower in practice than Iceberg's, which is precisely why the ecosystem strategy leans on translation layers. And mindshare momentum, fairly or not, has consolidated elsewhere, which affects hiring, tooling, and the long-tail integrations nobody plans for.

Apache Paimon: The Streaming Native

How it works. Paimon, which graduated to an Apache Top-Level Project in 2024 after beginning life as Flink Table Store, made the boldest architectural bet of the file-metadata formats: bring the log-structured merge tree - the storage design underneath databases like RocksDB - to the lakehouse. Primary-key tables organize data into LSM levels, writes land in sorted runs and compact downward continuously, and merge engines define what a key's latest state means - deduplicate, partial-update, aggregate - giving CDC semantics natively rather than as a bolted-on delete mechanism. Changelog producers emit both the table and its change stream simultaneously, and lookup joins let streaming jobs enrich against Paimon tables efficiently. Snapshots and time travel ride on top, so batch engines see a normal versioned table while streams see a living one.

Current state. Paimon 1.0 shipped in 2025 with the 1.0.x line continuing into 2026, and the project's two strategic moves this cycle are the tells: an Iceberg-compatibility mode that publishes Paimon tables with Iceberg-readable metadata, and participation in the REST catalog world - both acknowledging where read-side gravity lives while competing on write-side physics. The ecosystem is Flink-first and unapologetic about it, with Alibaba-scale production behind it, Spark support growing seriously, and a natural pairing emerging with Apache Fluss, the sub-second streaming storage layer that feeds Paimon tables for long-term state.