Apache Data Lakehouse Weekly: July 11 to July 18, 2026

Apache Iceberg

The most consequential thread of the week came from huaxin gao, who restarted the conversation on deprecating equality deletes, this time framed around the V4 spec. Russell Spitzer first proposed the idea back in October 2024, and the blocker then was real: Flink streaming upserts depended on equality deletes and had no practical replacement.

The new thread argues that the ground has shifted. Equality deletes are cheap to write but expensive to read, since every reader must join delete files against candidate rows across a sequence number range. Positional deletes and V3 deletion vectors read far faster - one compact bitmap per data file applied with an O(1) position check.

The thread goes past performance too. Equality deletes block CDC and row lineage because the true state of a table requires a full scan while they exist. They also force full rebuilds of materialized views and secondary indexes instead of incremental maintenance. Steven Wu, Manu Zhang, Maximilian Michels, and Xin Huang all weighed in, and the discussion drew eight messages in its first days.

For readers newer to Iceberg internals, the stakes are worth spelling out. Iceberg supports two ways to delete rows without rewriting data files:

• An equality delete says "remove every row where id equals 42," which a streaming writer can emit in microseconds without reading anything.
• A positional delete says "remove row 1,507 of file X," which requires the writer to know exactly where the row lives.

The first pushes all the cost onto readers, who must evaluate the predicate against huge swaths of data on every query. The second keeps reads fast at the price of more expensive writes. V3 deletion vectors compress the positional approach into one bitmap per data file. The V4 question is whether the format still needs the first option at all once conversion tooling makes the second cheap enough for streaming workloads.

The timing is not an accident. Maximilian Michels reported that the Flink equality delete to deletion vector conversion work is now complete, merged across six commits. The feature ships as a new table maintenance task called ConvertEqualityDeletes, integrated with IcebergSink. Writers keep producing data files and equality deletes as before. The converter reads them, resolves the deletes into deletion vectors using a primary key index backed by RocksDB in Flink state, and commits data files plus DVs to the target branch. Teams can stage writes on a separate branch so readers never see equality deletes at all, or run in-place conversion on a single branch. This is the workable alternative that was missing in 2024, and it lands right as the V4 deprecation push begins. Read the two threads together and you see a community clearing a path before it closes a door.

Release Work

Release work stayed busy. Danny Jones and Shawn Chang called a vote on Iceberg Rust 0.10.0 RC4 after RC3 gathered votes from Kevin Liu, Amogh Jahagirdar, and others earlier in the week. Sung Yun, Renjie Liu, L. C. Hsieh, and Maximilian Michels verified RC4, which includes a check that pyiceberg-core builds and tests cleanly against the release. The Rust implementation now sits underneath the Python ecosystem, so each Rust release carries weight well beyond Rust users.

That dependency chain is worth pausing on. PyIceberg increasingly delegates its performance-critical paths to pyiceberg-core, which is compiled from this Rust codebase. A bug in iceberg-rust becomes a bug in every Python notebook and Airflow DAG that touches Iceberg through PyIceberg. That is why the release checklist now explicitly verifies the Python bindings, and why voters from the Python side of the community show up on Rust release threads. The 0.10.0 line also continues the project's steady march toward feature parity with the Java reference implementation, which lowers the barrier for teams that want Iceberg without a JVM anywhere in the stack.

Infrastructure as code arrived as a first-class citizen this week. Matt Topol proposed RC0 of the Apache Iceberg Terraform Provider v0.1.0, the project's first release of the provider, with convenience binaries prepared for the Terraform and OpenTofu registries. Neelesh Salian, Alex Stephen, Sung Yun, Talat Uyarer, and Rich Bowen all participated in verification, and an RC1 vote followed as issues surfaced. Once this lands, teams can declare Iceberg resources in the same Terraform plans that manage the rest of their infrastructure.

Think about what that unlocks in practice. A platform team can define namespaces, tables, and their properties in version-controlled HCL, review changes through pull requests, and roll environments forward and back with the same tooling they use for VPCs and Kubernetes clusters. Catalog drift - the gap between what the catalog says and what the last runbook did - becomes a solved problem instead of a recurring incident. It took years for databases to get credible Terraform support. Iceberg is getting there in its first decade.

AI on the Dev List

AI showed up on the dev list in a very concrete form. Gang Wu asked the community about enabling ASF-managed GitHub Copilot code review on Iceberg repositories, starting with iceberg-cpp as a trial. The proposal uses the .asf.yaml setting that ASF Infra now supports, and it follows Apache Arrow, which enabled and tuned the same feature. Gang picked iceberg-cpp because reviewer bandwidth there is thin, and an automated first pass can catch simple issues before a human review. The thread drew eleven messages from Steve Loughran, Junwang Zhao, Scott Haines, and others, making it the most active Iceberg discussion of the week.

The questions were practical: what does it cost in CI resources, how noisy is the feedback, and who tunes it. There is a bigger question under the surface. Open source review is the mechanism by which projects transfer knowledge, enforce standards, and grow maintainers. An automated first pass that catches typos, missing null checks, and doc gaps frees human reviewers for design feedback, which is a clear win. An automated pass that contributors treat as the review, or that buries PRs in low-value comments, erodes the very culture it was meant to help. Starting with one low-traffic repository and evaluating before expanding is the right way to find out which outcome Iceberg gets. The fact that Arrow already ran this experiment and tuned it gives Iceberg a head start on configuration.

Type System Proposals

Two type system proposals advanced. Yan Yan opened a discussion on first-class vector type support for Iceberg. Embeddings are everywhere in AI workloads, and today they live in Iceberg as list, which cannot express the invariant that every value shares one dimension. The proposal prefers a dense numeric vector type with compact schema encoding, something like float[768], with fixed dimension, non-null elements, and nullability controlled at the field level. It points at parallel work in the Parquet community on fixed-size lists, which matters because the table format and the file format need to agree for the type to pay off.

Meanwhile the collation support discussion between Andrei Tserakhau, Alexander Löser, and Russell Spitzer dug into a genuinely hard question: how much cross-engine interoperability should the format guarantee for string ordering? Alexander laid out the trap in detail. ICU does not keep orderings stable across versions, so two engines on different ICU versions can sort the same strings differently, return different aggregation results, and filter different rows. A colleague of his once rolled back an ICU upgrade in production because users complained about changed sort orders. Pinning an ICU version at the table level buys consistency and costs upgrade freedom. The thread has not resolved the tension, and it is worth watching because collation touches execution, pruning, and equality semantics all at once.

File Format Layer

The file format layer got its own existential question. Martin Prammer proposed adding Vortex as an Iceberg file format, and smartly split the draft into two parts: what criteria any candidate file format should meet, and how Vortex meets them. That framing turns a single-format request into a durable policy, which is exactly what a spec-driven project needs as more formats knock on the door.

Quality and Correctness

Quality and correctness threads kept coming. Priyadarshini Mitra proposed a ValidateTableIntegrity action that walks the full metadata graph - metadata.json entries, manifest lists, manifests, data files, delete files including V3 deletion vectors, and statistics files - verifying every referenced file exists on storage. It supports a self-audit on one table and a source-versus-destination check for DR and migration scenarios, tracked across three sequential PRs.

Neelesh Salian, Sung Yun, and Andrei Tserakhau merged their earlier threads into one proposal for shared conformance fixtures, a standalone language-neutral repository modeled on parquet-testing, so every Iceberg implementation checks its reading of the spec against a shared answer key instead of only against itself. Working proofs of concept already exist for pyiceberg, iceberg-rust, and iceberg-go.

And Russell Spitzer moved to clarify in the spec that live manifest entries must be unique by file path, tightening language first added four years ago so writers know duplicate references are simply not allowed. Russell also raised a question about breaking behavior in AvroSchemaUtil, where adding LocalTimestamp support changes what convert returns for local-timestamp-micros, from Long to TimestampType.withoutZone(). His position: the old behavior is a bug, and Iceberg should not preserve incorrect legacy behavior behind a flag for outside consumers. Ryan Blue engaged on the thread, and the precedent cited is the earlier NanoTimestamp change that did the same thing.

Operational Updates

On the operational side, Oleksii Omhovytskyi asked about release timing for the encrypted deletion vector fix. On natively encrypted format-v3 tables, a merge-on-read UPDATE or DELETE wrote a deletion vector Puffin file without key metadata, and the next read failed. The fix is merged with backports staged on the 1.11.x and 1.10.x branches, and Oleksii verified the 1.11.x branch against his exact repro. He offered to test any release candidate - a nice example of a user pushing a patch release forward with evidence instead of just a request. Amogh Jahagirdar responded on timing.

Seeds Worth Tracking

Several single-message threads planted seeds worth tracking. Anurag Mantripragada proposed using Iceberg sort order metadata to improve read and compaction behavior in Spark. Iceberg tables already record their sort orders in metadata, but engines rarely exploit that knowledge at plan time, so there is free performance sitting on the table.

A CDC practitioner opened a thread on row-delta commit patterns and multi-table transactions in iceberg-rust, sharing lessons from a production change-data-capture pipeline and asking what the Rust library should support natively.

Oğuzhan Ünlü requested review on a PR adding typed exceptions for OAuth2 token endpoint errors in the API and core modules - small plumbing that makes REST catalog auth failures debuggable instead of mysterious.

Andrei Tserakhau also pitched a series of Iceberg technical blog posts with a first draft attached, and Matt Butrovich responded. Community-written deep dives are one of the best on-ramps a project can have, so this effort deserves support.

On the ecosystem edge, Piergiorgio Lucidi introduced the OpenCrawling connector, which bridges Iceberg tables into enterprise AI and RAG pipelines - one more signal that retrieval workloads now treat the lakehouse as a first-class source.

Rounding Out the Week

Rounding out the week: Adam Szita moved the KMS credential vending proposal into a draft REST OpenAPI spec PR, mirroring storage credential vending so REST catalogs can return short-lived scoped key-management credentials for encrypted tables. Ryan Blue weighed in on Spark routing for Iceberg Materialized Views, favoring a basic implementation in Iceberg itself that replaces a view with a table read, usable without engine APIs, with engines layering smarter freshness decisions on top over time.

Alexandre Dutra opened a discussion on migrating Iceberg to Jackson 3, a mechanical but pervasive change whose hardest part is that Jackson types leak into the public API of the parser classes and REST layer, so Jackson 2 and 3 will need to coexist for a while. A contributor from Alibaba proposed adding an Alibaba Cloud auth type for the REST catalog. And Talat Uyarer announced an Apache Iceberg meetup in Austin on July 23.

Apache Polaris

Polaris had the busiest list of the six projects this week at 84 messages, and the center of gravity was persistence consistency. Dmitri Bourlatchkov opened a discussion...