Iceberg catalogs explained: REST, Glue, S3 Tables, Polaris, Nessie

"Iceberg catalog" sounds like a big piece of infrastructure: a metadata service, a governance layer, something with a vendor attached. The core job is much smaller.

A catalog keeps one pointer per table, to the table's current metadata.json, and changes that pointer atomically when someone commits. Everything else about the table lives in files.

Once you see that pointer, the choice between REST, Glue, S3 Tables, Polaris and Nessie gets easier. They differ in what they add around the pointer, and in which engines can reach them.

I have not run Iceberg in production. To show the catalog concretely, I built a table on my laptop with PyIceberg 0.12.0 and DuckDB 1.5.6, using a SQL catalog on SQLite and then on a throwaway Postgres 16. The rows below are from those runs.

What is an Iceberg catalog?

An Iceberg table is a folder of Parquet data files plus a tree of metadata. The top of the tree is a metadata.json holding the schema, the partition spec and the list of snapshots.

Every commit writes a new metadata.json. Nothing is edited in place. So after 50 commits there are 51 of them, and something has to say which one is current. That something is the catalog.

Here is the entire state of PyIceberg's SQL catalog after I created one table and appended 50 batches to it. The catalog database held two tables:

• iceberg_tables
• iceberg_namespace_properties

And iceberg_tables held one row:

• catalog_name lab
• table_namespace shop
• table_name events
• metadata_location .../events/metadata/00050-c499c071-....metadata.json
• previous_metadata_location .../events/metadata/00049-bb259239-....metadata.json
• iceberg_type TABLE

That is it. One row, two file paths. The table's schema, its 50 snapshots and its file lists are all in the 50 KB metadata.json that row points to. The catalog holds a few hundred bytes.

A commit is a compare-and-swap

When a writer commits, it writes new data and metadata files, then asks the catalog to move the pointer. The SQL catalog does it with one statement. Simplified from PyIceberg's source:

UPDATE iceberg_tables SET metadata_location = '.../00051-....metadata.json', previous_metadata_location = '.../00050-....metadata.json' WHERE table_namespace = 'shop' AND table_name = 'events' AND metadata_location = '.../00050-....metadata.json'; -- the version I started from

If another writer committed in between, the WHERE matches no row, and PyIceberg raises CommitFailedException. I tested it on Postgres: two handles on the same table, both appending. The second append conflicted, PyIceberg retried 108 ms later on top of the new state, and both rows landed.

That check is why the catalog exists. Object storage has no cheap way for two writers to agree on which file is "current". The catalog gives them one place to take turns.

After a delete, a schema change, an append, a rewrite and a snapshot expiry, the same row pointed at 00055-....metadata.json, with 00054 as the previous one. Every operation moved the pointer once.

This is the first part. The full post - including the rest of the working details - is on my site: Iceberg catalogs explained: REST, Glue, S3 Tables, Polaris, Nessie