Auto CDF Moved the CDC Tax From Writes to Reads: Delta CDF vs Auto CDF vs Spark 4.2 CHANGES
DEV Community

Auto CDF Moved the CDC Tax From Writes to Reads: Delta CDF vs Auto CDF vs Spark 4.2 CHANGES

On September 1, 2026, Databricks made automatic change data feed (Auto CDF) generally available. The release note says it "removes write-time overhead and makes MERGE and UPDATE operations about 15% faster on tables that you query for changes."

Seven weeks earlier, Apache Spark 4.2.0 (July 14) added a SQL CHANGES clause, so reading row-level changes is now a first-class part of the engine, not a Delta-only option.

Most of the takes I have seen stop at "free CDC, faster MERGE, turn it on." My thesis is narrower: Auto CDF does not remove the cost of change data. It moves it from the writer, who pays once per commit, to every reader, who pays on every read. Its prerequisite, row tracking, has a write-time cost of its own that nobody puts on the slide.

Spark 4.2's CHANGES is a contract, not an implementation. Whether it works depends on your connector, and on open-source Delta 4.4.1 it does not work out of the box. So the real question is not "legacy or auto." It is: how many times is each commit read, and how much data does each commit rewrite?

Three ways to get the changes

Legacy CDF (write time) / Auto CDF (read time) / Spark 4.2 CHANGES

  • Where the work happens
    • Writer writes change files next to the data
    • Reader computes changes from row tracking metadata
    • Connector supplies a changelog, Spark post-processes
  • Opt in
    • delta.enableChangeDataFeed = true per table
    • Automatic when requirements are met
    • Catalog must implement loadChangelog
  • Who can read it
    • Any Delta reader that implements the cdc action of the protocol
    • Databricks readers only (DBR 19+)
    • Any Spark 4.2 client, if the connector supports it
  • Formats
    • Delta only
    • Delta with row tracking, Iceberg v3
    • Whatever the connector exposes
  • Read API
    • readChangeFeed, table_changes()
    • Same APIs
    • SELECT ... CHANGES FROM VERSION, .changes()

FACT (Databricks docs): legacy CDF "materializes changes during table writes", Auto CDF "computes changes during table reads using row lineage metadata", and the two cannot be used together on one table.

Migration is a single statement:

ALTER TABLE t UNSET TBLPROPERTIES ('delta.enableChangeDataFeed')

The mechanism, in two paragraphs

Legacy CDF. The Delta protocol has a cdc action. On UPDATE, DELETE and MERGE the writer puts the changed rows, with _change_type, into files under _change_data/. The protocol tells readers to use those files when they exist and skip the add and remove actions for that version. Insert-only commits and full partition deletes produce no change files, because the log already says what happened. So the reader of a commit touches roughly the changed rows, nothing more.

Auto CDF. Row tracking gives each row a stable row ID and a row commit version. The Delta protocol explains how: for inserts, the IDs are derived from a baseRowId per file plus the row position, which costs almost nothing. But when a writer copies a row into a new file (the "copy" in copy-on-write), it must preserve the stable ID, so it materializes the row ID and commit version into hidden columns in the data file. At read time, the engine compares row IDs and commit versions across the files a commit removed and added to work out which rows were inserted, updated or deleted. That is real computation over real files, and it happens on every read.

What I measured

I cannot run Databricks' Auto CDF outside Databricks, and I am not going to pretend otherwise. What I can run is open-source Delta Lake 4.4.1 on Spark 4.2.0, which has both halves of the mechanism: legacy CDF, and row tracking with _metadata.row_id and _metadata.row_commit_version.

Setup: A 10M-row Delta table (about 507 MB of Parquet), deletion vectors off, so MERGE is pure copy-on-write. Each round: a MERGE that updates about 100,000 rows (1%, keys hashed across the whole id range) and inserts 10,000 new ones. 5 rounds per variant, medians reported, after a discarded warm-up. Three variants: plain, legacy CDF, row tracking only.

Read side: read the changes of the last commit. For legacy CDF that is readChangeFeed for one version. For row tracking it is a reconstruction I wrote: read the files the commit removed (at version v-1) and added (at version v), and join on row_id and row_commit_version. I checked the reconstruction against legacy CDF on a table with both features enabled: for a MERGE (200 inserts, 1,986 pre-images, 1,986 post-images) and for a DELETE (100 deletes) the counts matched exactly.

The core of the reconstruction:

after = (
  spark.read.format("delta").option("versionAsOf", v).load(path)
  .select("*",
    F.col("_metadata.row_id").alias("rid"),
    F.col("_metadata.row_commit_version").alias("rcv"),
    F.col("_metadata.file_path").alias("fp"))
  .where(F.col("fp").isin(added_files))
)
changed = after.where(F.col("rcv") == v) # inserts + post-images
# pre-images: rows in removed files whose rid appears in `changed`
# deletes: rows in removed files whose rid appears nowhere in `after`

Results, 10M rows, scattered 1% upsert, median of 5 rounds:

  • Variant MERGE time / Bytes written per MERGE / Change files per MERGE / Read one commit's changes
  • Plain 6.03 s (4.75 to 7.69) / 505.8 MB / none / n/a
  • Legacy CDF 6.52 s (6.19 to 7.10) / 506.0 MB / 10.8 MB / 0.21 s
  • Row tracking only 6.16 s (5.92 to 7.69) / 547.6 MB / none / 4.73 s (my reconstruction)

A second, partial run of the plain and legacy CDF variants gave 5.64 s and 6.89 s for the MERGE, and 0.18 s for the CDF read.

What the numbers say, and what they do not

  1. The write tax of legacy CDF is real but small here, and smaller than my noise. Legacy CDF was slower in both runs (6.52 vs 6.03 s, and 6.89 vs 5.64 s), but the spread inside one variant (4.75 to 7.69 s) is wider than the gap. Databricks measured about 15% on its runtime; my numbers are compatible with that and cannot confirm it. The byte count is solid: 10.8 MB of change files per MERGE, about 2% of what the MERGE rewrote anyway.

  2. Row tracking wrote 8% more data on every rewrite. 547.6 MB versus 505.8 MB per MERGE, because every copied row now carries its stable row ID and commit version in hidden columns. That is about four times the size of the CDF change files in this workload. This is the part of the "free" read-time CDF that you pay at write time.

  3. Reading changes at read time costs what the commit rewrote, not what it changed. The 1% scattered upsert touched every file, so the commit removed and re-added the whole table. Legacy CDF read 10.8 MB of change files in 0.21 s. My reconstruction had to scan both sides of a 500 MB rewrite to find 100,000 changed rows: 4.73 s, about 22 times slower. On a smaller test (200,000 rows, 8 files) the same scattered upsert rewrote 8 of 8 files, while an upsert clustered on the newest ids rewrote 1 of 8. The layout of your updates decides how expensive read-time CDF is.

What I cannot claim: that Databricks' implementation is as slow as mine. It is very likely smarter, and with deletion vectors a MERGE writes only the changed rows to new files and marks the old ones, which shrinks the set of rows a reader has to compare. My test is the copy-on-write case, which is the worst case for read-time CDF. The mechanism still holds: the read-time cost scales with what a commit touched, and you pay it once per reader per read.

  1. CHANGES on open-source Delta 4.4.1 fails by default. With the stock DeltaCatalog, all three of SELECT * FROM t CHANGES FROM VERSION 1, the path-based form, and spark.read.changes("t") returned:
[UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE] The feature is not supported: Catalog spark_catalog does not support Change Data Capture (CDC). SQLSTATE: 0A000

That matches the source. Delta PR #6794 (merged May 2026) adds catalog-driven read-time CDF behind an internal flag that is off by default, requires row tracking, and leaves the streaming side of the new path as a follow-up. I did not test the flag; it is internal and I would not run production on it. Spark 4.2 defines the shape of change data. Your connector decides whether you get any.

Gotchas to check before you run UNSET

All FACT, from the Databricks and Spark docs:

  • Auto CDF is not supported on tables with row filters or column masks. If your sensitive tables are governed with ABAC or masks, they are exactly the ones that cannot move.
  • Only Databricks readers can query Auto CDF on Delta, and external Iceberg clients cannot query it on Iceberg v3. Legacy CDF lives in the open Delta protocol. If anything outside Databricks reads your change feed, check before migrating.
  • It needs Databricks Runtime 19 or above and a table registered in Unity Catalog with row tracking enabled (or Iceberg v3). Enabling row tracking on an existing table is not free. It assigns IDs to every existing row, can create several new table versions, and concurrent writers fail with MetadataChangedException while it runs. Pause streaming writers first. It also upgrades the writer protocol, so older external Delta clients that do not support those features can no longer write to the table.
  • Multi-statement transactions: Auto CDF is not supported if the source table was modified during the transaction.
  • _metadata.row_id and row_commit_version cannot be read while reading the change feed. If your downstream logic wants both, plan for it.
  • Spark 4.2 streaming netChanges holds back the latest commit. Rows touched only in the latest commit are not emitted until a later commit moves the watermark, and streaming net changes can differ from batch net changes over the same range. Fine for a pipeline, surprising for a low-latency consumer.

Decision table

OPINION, built on the mechanism and my numbers:

  • Your table looks like: Write-heavy, changes read rarely or by one pipeline / Better choice: Auto CDF / Why: You stop paying on every write for a feed few people read
  • Your table looks like: One commit read by many consumers, often / Better choice: Legacy CDF, or Auto CDF plus one stream that lands changes in a table / Why: Read-time cost multiplies by readers times reads
  • Your table looks like: Scattered updates on a copy-on-write table / Better choice: Legacy CDF, or enable deletion vectors and measure / Why: Each commit rewrites most files, so read-time diffing is expensive
  • Your table looks like: Row filters or column masks / Better choice: Legacy CDF / Why: Auto CDF is not supported there
  • Your table looks like: Change feed read outside Databricks / Better choice: Legacy CDF / Why: Auto CDF is Databricks-readers only
  • Your table looks like: Iceberg v3 tables on Databricks / Better choice: Auto CDF / Why: Legacy CDF does not support Iceberg

What I would do this week

Inventory who reads each change feed. Count consumers and their trigger intervals per table. That count is the multiplier on read-time cost, and nobody has it written down.

Look at the update layout. In table history, compare numTargetRowsUpdated against files rewritten for your MERGE operations. A 1% update that rewrites 100% of files is a bad candidate for read-time CDF.

Migrate the boring tables first: write-heavy, one reader, no masks, nothing outside Databricks. Keep legacy CDF on the hot, many-reader tables until you have measured both on your own runtime.

If you fan out, fan out once. One stream reads the change feed and writes a changes table; the other consumers read that. The Databricks docs already show this archive pattern for permanent history.

Auto CDF is a good default for most tables, and a quiet bill for the ones read by five pipelines. CDC is still a cost. The release moved it to the reader, and the bill now goes up with every pipeline that reads the feed.

Hardware & versions

Box: 8 vCPU Intel Xeon VM, 16 GB RAM, Linux 6.12, OpenJDK 21.0.12. PySpark 4.2.0, delta-spark 4.4.1 (delta-spark_4.2_2.13), local[8], driver memory 8 GB (5 GB in the partial second run), 16 shuffle partitions. 10M rows, 5 columns, about 507 MB Parquet, deletion vectors off. 5 rounds per variant, medians, warm-up discarded. Same box also hosts another long-running benchmark of mine. During the main run its runner was paused waiting for an idle host, but its services stayed in memory. The partial second run overlapped that benchmark's active phase, so I quote it only as a cross-check. Treat timings as indicative. The byte counts and the correctness check are deterministic. Measured October 26, 2026. Script: cdf_bench.py (available on request).

Sources

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.