Apache Data Lakehouse Weekly: July 1 to July 8, 2026

Apache Parquet Adopts Versioned Releases for Breaking Changes

The lakehouse community spent this week deciding how change itself should work. Apache Parquet opened a formal vote to adopt versioned releases for breaking changes, borrowing a governance model that Iceberg refined over years.

Apache Polaris canceled a vote on its semantic model API so it can align with Apache Ossie, the freshly incubating semantics project that opened its dev list this week. And across Iceberg, Arrow, and Parquet, contributors debated who owns statistics, which format features deserve to survive, and how far the specs should bend to serve AI and machine learning workloads.

The connective tissue this week is governance. These communities are building the rules for evolving open formats without breaking the millions of tables that already depend on them. That work is invisible when it goes well, which is exactly why it deserves a close read while it happens.

Releases also kept pace with the design debates: Polaris shipped 1.6.0, Arrow Rust shipped 59.1.0, and both Iceberg Rust 0.10.0 and Arrow 25.0.0 entered their final voting rounds.

Apache Iceberg

The Rust implementation dominated release activity this week, and the path was not smooth. For readers who mostly touch Iceberg through Spark or a query engine, iceberg-rust deserves a moment of framing. The Java implementation remains the reference, but the Rust library has become the foundation for a second generation of tooling: pyiceberg-core binds it into Python, DataFusion integrates it for query processing, and a wave of lightweight services use it to read and write tables without a JVM. When this library ships a bug, the blast radius crosses language ecosystems, which explains the ceremony you are about to read.

Danny Jones and Shawn Chang opened the vote for Iceberg Rust 0.10.0 RC2 on July 1, drawing verification from L. C. Hsieh, Matt Butrovich, Neelesh Salian, Renjie Liu, and Xin Huang across a twelve message thread. Issues surfaced during verification, and by July 8 Jones was back with a vote on RC3. The willingness to cut a third candidate rather than wave through a flawed second one says something about how seriously the Rust community takes its release checklist, which covers everything from ASF license headers to a clean build of the pyiceberg-core bindings. The Rust library now sits underneath a growing stack of Python and query engine integrations, so the caution pays forward.

Statistics Ownership Debate

The week's most consequential design debate concerned table statistics, a topic that sounds dry until two engines start fighting over the same files. Some background helps here. Query engines rely on statistics - things like distinct value counts and data distributions - to choose good execution plans. Join two tables in the wrong order and a query that should take seconds takes minutes, so statistics quality translates directly into compute cost.

Iceberg lets engines write statistics files and attach them to a snapshot in table metadata. That design works cleanly when one engine owns a table. The trouble starts in the multi-engine deployments Iceberg was built for, where Spark handles ingestion, Trino or Dremio serves interactive queries, and Impala or Flink sits somewhere in the mix, each with its own idea of which statistics it wants.

Dzeri96 raised concerns about how Iceberg handles existing statistics files, and Gábor Kaszab pushed back on several proposed remedies across a nine message exchange. The core problem: when engine A writes statistics for a snapshot and engine B later computes its own, B can overwrite A's work.

One proposal would create a new snapshot whenever statistics get computed. Kaszab argued this breaks the mental model, since snapshots today mark data changes, not metadata additions, and a snapshot X holding stats for snapshot Y confuses more than it clarifies. Another proposal would bind multiple statistics files to one snapshot, keyed by engine. Kaszab questioned how readers match files to engines in practice. Does Impala version X.Y know whether it can read stat files written by Spark version A.B? Would engine IDs live in the Iceberg spec or in tribal knowledge?

The thread closed the week without resolution, which is fine. Multi-engine interoperability is Iceberg's whole reason for existing, and the statistics gap is a real hole in that story that deserves a careful fix rather than a fast one.

Security and Access Control

Security and access control produced a second substantial discussion. A quick primer for readers newer to the REST catalog: when a client asks the catalog for a table, the catalog can vend temporary storage credentials scoped to that table, so the client reads data files directly from object storage without holding permanent cloud keys. This delegation model is one of the REST catalog's best ideas, since it centralizes access decisions in the catalog while keeping the data path fast.

William Hyun proposed extending it with file-level access delegation, with Kevin Liu joining the exchange. Today, delegated access in the REST catalog operates at table scope, and that granularity is the limitation Hyun wants to fix. If a consumer should only see a subset of partitions, administrators either over-provision access or fragment tables to match access boundaries, and both options create operational pain.

Hyun's proposal uses pre-signed URLs during scan planning to make partition-scoped sharing practical without restructuring tables. Concretely, a catalog planning a scan for a restricted consumer returns signed links only for the files that consumer is entitled to see, and the storage layer enforces the boundary because unsigned paths simply fail. The idea builds on an earlier delegation thread and moves Iceberg closer to the fine-grained sharing models that commercial platforms offer on top of the format.

That last point carries strategic weight. Fine-grained access control is one of the few remaining capabilities where proprietary lakehouse platforms hold a clear edge over the open spec, and standardizing it in the REST catalog narrows that gap for every open implementation at once. Nevin Zheng added a related thread on updating the Read Restrictions proposal to adopt the Expressions Spec and IDReference, which points toward a more unified security model across these proposals.

Primary Key Conversations

The primary key conversation resurfaced with new energy. Anyone coming from the database world finds this gap surprising, so it deserves a plain statement: Iceberg tables have no primary keys. The format grew up serving analytical scans over immutable files, where keys matter less, and it handles row-level changes through delete files rather than key-addressed updates.

That works, but change data capture pipelines - which replicate every insert, update, and delete from an operational database into the lakehouse - spend enormous effort reconstructing key semantics the format never promises. Chandra Sekhar K continued the discussion on first-class primary key tables, describing production experiments with primary-key-oriented semantics for CDC and mutable-data workloads.

His team treats the primary key as table metadata and then builds behavior on top of it: key-aware write semantics, storage organization, compaction strategy, and changelog generation for incremental processing. He framed this as complementary to the existing constraints proposal, which limits PRIMARY KEY and UNIQUE to informational metadata and leaves enforcement out of scope. The distinction matters because it sketches a two-layer future. The spec standardizes how keys are declared, and engines opt into richer key-aware behavior above that line.

Anyone watching the upsert and CDC space, where formats like Apache Hudi and Paimon built key handling in from the start, should track this thread closely. A related note from st...@steveis.com asked about row-delta commits and multi-table transactions in iceberg-rust from a CDC producer's perspective, showing the same pressure arriving through the Rust door.

Geospatial Work

Geospatial work is quietly becoming one of Iceberg's most active frontiers. Sunmin Lee proposed declaring row-level bounding box covering columns in Iceberg metadata, which lets readers prune data files using spatial bounds. Seyed Muhammad Mahdi Hoseini followed with a separate proposal for QuadTree-inspired physical spatial partitioning, with Tanmay Rauth engaging on the bbox thread.

Together these ideas sketch a spatial stack for the format: partition data by spatial cells on write, prune by bounding boxes on read. The economics mirror what min-max statistics did for numeric columns years ago. A query asking for events inside a city polygon should never open files whose contents sit on another continent, and today that pruning depends on engine-specific tricks rather than portable metadata.

Iceberg v3 introduced geometry and geography types, so the type system groundwork exists, and these proposals show the community moving from type support to the performance engineering that makes spatial workloads economical. Fleet telemetry, logistics, climate data, and location-based applications all stand to benefit, and they represent exactly the data volumes where file pruning changes the bill.

Variant Type Progress

The variant type effort keeps a steady drumbeat. Variant is the shredded, binary-encoded type for semi-structured data that lets JSON-shaped payloads live in tables with columnar performance, and it spans both the Iceberg and Parquet specs, which makes coordination the hard part. A field promoted to a shredded column by one writer has to remain readable by every engine, and the type touches file format, table format, and engine layers at once.

That is why the community runs a dedicated sync for it. Neelesh Salian posted notes and a recording from the July 2 Variant Sync, the recurring gathering that coordinates work on semi-structured data support across implementations, with action items now tracked in a shared document. The cadence of these syncs since April shows the feature moving through the unglamorous middle stage of standardization, past the exciting design documents and into the grind of cross-implementation agreement.

Anurag Mantripragada opened a related design question about column update file representation, which touches how partial column updates get written to storage.

Spec Hygiene

Spec hygiene rounded out the Iceberg week. Daniel Weeks opened threads on clarifying schema JSON type string serialization and clarifying valid source types for the identity transform. Sung Yun raised a write-path gap for field-id-bound policy during schema evolution and a question about field id handling in the REST spec.

These threads rarely make headlines, but they are the reason independent implementations of Iceberg in Java, Rust, Python, Go, and C++ produce the same answers on the same tables. Every ambiguity closed on the list is a bug that never ships, and the current volume of clarification work reflects the v3 and v4 feature waves pushing the spec into corners nobody had to define precisely before. Spec text that a single reference implementation can leave fuzzy becomes load-bearing the moment a second implementation reads it differently.

Additional Items

A few more items deserve a sentence each:

• Szehon Ho called a vote to add specific-name to the UDF spec, advancing the effort to make user-defined functions portable across engines.
• Alexander Löser continued the collation support discussion, which matters for anyone whose sort order needs to survive engine boundaries.
• Daniel Weeks flagged an upcoming Iceberg Terraform provider release with Alex Stephen, bringing infrastructure-as-code workflows to table management.
• Tanmay Rauth proposed a table_properties_log metadata table to expose property history for audit and debugging.
• Renjie Liu suggested trimming CI runner time by running JDK 21 checks only on main and nightly builds, a small change that respects contributor time on every pull request.

Apache Polaris

The Polaris story of the week is a vote that did not happen, and the cancellation matters more than most votes that pass.

First, the context. A semantic model defines business concepts - metrics like revenue or monthly active users, dimensions like region or customer segment, and the logic connecting them to physical tables. Today those definitions live scattered across BI tools, dbt projects, and application code, and every copy drifts from the others. The industry calls the result semantic drift, and it is why two dashboards in the same company report different numbers for the same metric.

The Open Semantic Interchange effort, backed by a broad vendor coalition, produced a specification for expressing these definitions in a portable, machine-readable format. Catalogs are the natural home for such definitions, since they already serve as the shared source of truth for tables, and that is where Polaris comes in.

Yufei Gu had called a vote to accept the OSI Semantic Model API Specification, the proposal that would let Polaris store and serve business metric definitions alongside its catalog duties. Jean-Baptiste Onofré asked the community to pause. His reasoning: the Open Semantic Interchange specification is transitioning into the Apache Ossie project, which entered incubation on June 22, and an initial Ossie spec release is expected soon. Implementing the OSI spec now and the Ossie spec later means doing the work twice.

Robert Stupp and Adam Christian weighed in across the thirteen message thread, and Gu canceled the vote after gathering the feedback. He noted the practical middle path along the way: most of the semantic model work in Polaris can proceed in parallel, since the main dependency on Ossie is a JSON validator that arrives later, and the API can ship marked as beta with an explicit plan to converge on the Ossie specification.

Gu then did exactly that kind of parallel work, opening a focused design thread on the semantic model REST API payload representation. The question sounds technical but shapes years of API stability: should Polaris represent...