The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough

The Physics: Why Iceberg Has a Freshness Floor

Everything in this article follows from three facts about how Iceberg works. Get these into your head and every product claim becomes easy to evaluate.

Fact one: data is invisible until committed. An Iceberg table is defined by its metadata. A writer can push Parquet files to object storage all day, but no query sees a single row until a commit publishes a new snapshot that references those files. The commit is the moment of visibility. So end-to-end freshness is never just "how fast can I write data." It is "how often do I commit," plus the time the data spent buffering before the commit, plus the time a query engine takes to notice the new snapshot. Any latency number that ignores the commit cadence is marketing.

Fact two: commits are not free. Every commit writes new metadata - a fresh metadata file, manifest list, and manifests under the current v3 format - and performs an atomic swap through the catalog. That work takes real time against object storage, typically measured in seconds, and concurrent committers to the same table contend and retry. This creates a practical floor. Commit every second and metadata work dominates, the catalog becomes a chokepoint, and snapshot history balloons. The workable range for commit intervals today runs from a few seconds at the aggressive end to minutes at the comfortable end. This is exactly the write amplification problem that the Iceberg v4 proposals around single-file commits and adaptive metadata trees are designed to shrink, and it is worth knowing that help is coming at the format level. But you are building on v3 today, and on v3 the floor is real.

Fact three: frequent commits create small files, and small files poison reads. Commit every ten seconds and you produce 8,640 commits a day, each adding files sized by whatever trickled in during those ten seconds. Thousands of tiny Parquet files mean thousands of object store requests per query, bloated metadata, and planning that slows week by week. Streams with updates and deletes add delete files or deletion vectors on top. The only cure is maintenance: compaction to merge small files and snapshot expiration to trim history. Every streaming pipeline into Iceberg is therefore really two pipelines - the ingestion path and the maintenance path - and the most common failure I see in the field is deploying the first without the second. Freshness you cannot query is not freshness.

Hold those three facts and the whole vendor bake-off becomes legible. Every option below is a different answer to the same three questions: who buffers the data, who decides when to commit, and who runs the maintenance.

The Open Source Workhorses

Start with the three options that dominate real deployments, all open source, all mature, each occupying a distinct point on the latency-versus-effort curve.

Apache Flink: the low-latency standard

Flink is the reference answer for the lowest-latency open path into Iceberg, and it earned that position. A Flink job consumes from Kafka or another source, processes events continuously, and writes to Iceberg with commits tied to Flink's checkpoint cycle. Checkpoints align with Iceberg snapshot commits, which is what gives Flink exactly-once delivery into the table: either a checkpoint completes and its data becomes a committed snapshot, or neither happens.

Latency lands wherever you set the checkpoint interval, and in practice that means seconds. Teams commonly run 10 to 60 second checkpoints against Iceberg, with aggressive setups pushing toward the low end of that range. Between commit floor physics and checkpoint overhead, think of well-run Flink-to-Iceberg freshness as roughly ten seconds to a minute, with the aggressive end paying more maintenance tax.

Flink's second superpower is change data capture. Flink CDC connects directly to database transaction logs, and Flink writes changelog streams - inserts, updates, and deletes - into Iceberg using equality deletes for the rows it cannot cheaply locate. This makes Flink the standard tool for maintaining a near-real-time Iceberg mirror of an operational database. The cost of that convenience is the equality delete backlog, which slows reads until maintenance resolves it. The good news in 2026 is that this exact pain is getting format-level and engine-level attention: v3 deletion vectors made positional deletes cheap to read, and work is active in the community on having Flink convert equality deletes into deletion vectors closer to write time, shrinking the window where readers pay the matching tax.

The trade-offs are operational. Flink is a distributed stateful system that you must size, checkpoint, upgrade, and debug, and Flink expertise is its own hiring line. Small file pressure is high at short checkpoint intervals, so the maintenance pipeline is mandatory. Choose Flink when you need seconds-level freshness, exactly-once guarantees, CDC semantics, or in-stream transformation, and you have - or can rent - the operational muscle. Managed Flink offerings from Confluent, AWS, Ververica, and Decodable exist precisely for teams that want the engine without the pager duty.

Spark Structured Streaming: the pragmatic middle

Spark Structured Streaming writes to Iceberg on a micro-batch model. You set a trigger interval, Spark accumulates data, and each trigger produces one Iceberg commit. A 60 second trigger yields 1,440 commits a day, each with reasonably sized files. Freshness lands at seconds to minutes depending on the trigger, with a minute being the comfortable default.

The case for Spark is continuity. Most data teams already run Spark for batch, already know its APIs, and already operate its clusters. Adding a streaming ingestion job is an increment, not a new platform. The trigger interval gives you a single clean dial between freshness and file health: lengthen it and files fatten, shorten it and freshness improves. Spark also brings the full transformation library to the stream, and the same job pattern serves backfills.

The case against is the latency ceiling and the cost profile. Micro-batching means Spark will not chase Flink to the aggressive end of the freshness range, and always-on streaming clusters are frequently overprovisioned, with a large share of allocated compute idling between triggers. Choose Spark when a minute of freshness is fine and your team is already a Spark team. That describes a lot of teams, which is why this path is more common in practice than the discourse suggests.

The Iceberg Kafka Connect sink: the no-code path

The community-maintained Iceberg sink connector for Kafka Connect reads topics, buffers records, and commits to Iceberg on a configurable interval, with a control-topic mechanism coordinating commits across connector tasks so the table gets clean, consistent snapshots. It supports automatic table creation, schema evolution driven by Schema Registry, and routing records to different tables or partitions. There is no application code. You deploy a JSON configuration into a Kafka Connect cluster, and Connect handles scaling, offsets, and fault recovery.

For an organization already running a Connect estate for other sinks, adding Iceberg is an afternoon. Freshness lands in the minutes range at typical configurations, since the connector's economics favor fewer, larger commits, and that gentler commit cadence is also why its small-file pressure runs lower than aggressive Flink or Spark setups.

The trade-offs: latency is minutes, not seconds; transformation capability is limited to lightweight single-message transforms; and CDC upsert flows are less natural here than in Flink, though supported patterns exist. And if you do not already run Kafka Connect, standing up a Connect cluster just for this erases much of the simplicity argument. Choose the sink when you have Kafka, you have Connect, minutes are acceptable, and you want the pipeline nobody has to babysit.

The Broker-Native Wave: When Kafka Itself Writes Iceberg

The most interesting structural development of the past two years is the collapse of the pipeline itself. A generation of Kafka-compatible platforms now materializes topics as Iceberg tables from inside the broker layer - no Flink job, no Connect cluster, no separate ingestion service. The pitch is "stream once, query forever," and the implementations differ in ways that matter.

• Redpanda's Iceberg Topics persist topic data directly into Iceberg format from the broker, with automated housekeeping like snapshot expiration and custom partitioning. Time-to-value is excellent, since enabling a table is a topic-level switch.
• StreamNative's Ursa engine takes a lakehouse-native approach, storing data through a write-ahead log for ingestion and Parquet for analytics, speaking the Kafka protocol on a leaderless architecture.
• AutoMQ, a stateless S3-native Kafka, offers table topics with a similar promise, and pairs it with query-time federation ideas aimed at hiding the gap between not-yet-committed stream data and committed table data.
• Bufstream targets Protobuf-heavy shops with schema governance built in and Iceberg as the landing format.

Now the honest physics. Moving the Iceberg writer into the broker does not repeal the commit floor. These systems still buffer, still write Parquet, still commit snapshots, and the freshness of the queryable table still lands in the seconds-to-minutes range depending on configuration - typically closer to minutes at sane settings. What broker-native designs actually buy you is the removal of an entire operational tier, and that is genuinely valuable.

What they can cost you shows up in the fine print. Some zero-copy designs that make Iceberg the broker's primary storage push producer latency up several fold versus classic Kafka, since produces now ride object storage economics. Some produce tables that are effectively read-only from the outside, or that lack automatic compaction and snapshot hygiene, which quietly hands the maintenance pipeline back to you. And Kafka protocol compatibility varies at the edges - transactions and compacted topics being the classic gaps - which matters if the rest of your stack assumes full Kafka semantics.

My guidance: broker-native Iceberg is the right default for append-only event streams landing in a lakehouse when you are already choosing one of these platforms for other reasons. Interrogate three things before committing: producer latency impact, who runs compaction and expiration, and whether the resulting tables are first-class citizens that any engine can maintain and evolve, or a materialized view you can only look at.

The Managed Layer: Vendor Pipelines and Cloud Services

Above the open source and broker options sits a thick layer of managed offerings, and in 2026 this is where most net-new pipelines I encounter actually get built. A tour of the ones that come up most.

Confluent's Tableflow is the highest-profile entry: a checkbox-level feature in Confluent Cloud that materializes Kafka topics as Iceberg tables, handling schema mapping, type conversion, file sizing, compaction, and catalog publication, with Delta Lake as an alternate output and integration into Confluent's governance stack. Paired with Confluent's managed Flink for in-stream transformation, it forms a complete stream-to-lakehouse path where you never touch a cluster. The trade is the classic managed trade: meaningful cost at scale and deep attachment to one vendor's ecosystem. Freshness is in the minutes class, governed by its materialization and compaction cadence.

WarpStream, under the Confluent umbrella, offers its own TableFlow with a bring-your-own-cloud model that can source from any Kafka-compatible cluster, trading some polish for openness and cost control.

On AWS, the native path got legitimately good. Kinesis Data Firehose delivers streams directly into Iceberg tables with buffering measured in tens of seconds to minutes, and S3 Tables provide Iceberg storage with built-in automatic compaction and snapshot management, which removes the maintenance pipeline that self-managed teams forget. Glue and EMR cover the Spark and Flink routes with v3 support. The result is a fully AWS-native stream-to-Iceberg story with freshness in the one-to-several-minutes class and very little to operate, at the price of AWS coupling.

Snowflake's Snowpipe Streaming writes row-level streams into Snowflake-managed Iceberg tables with seconds-to-minute visibility, and those tables remain readable by external engines - a meaningful step toward interoperability from a vendor that historically kept data inside its walls.