The File Format Renaissance: Parquet, Lance, Vortex, Nimble, BtrBlocks, and the New Physics of Columnar Storage

First Principles: What a File Format Actually Decides

Strip the category to its decisions, because every format in this article is a different set of answers to the same five questions.

How are values laid out? Columnar versus row-oriented is the famous decision, and within columnar, the finer ones: how rows are grouped, whether groups are fixed or adaptive, how nested and variable-length data is represented.

How are values encoded? The compression stack, from lightweight structural encodings - dictionary, run-length, delta, bit-packing - to heavyweight general codecs like Zstandard, chosen per column or per chunk, chained or singular.

What metadata travels with the data? Schemas, statistics, offsets, indexes - the self-description that lets readers plan before reading.

How is data accessed? Optimized for sequential scans, for random point access, for both, and at what granularity readers can retrieve without touching neighbors.

And what is the contract? A byte-level specification anyone can implement, or a library whose API is the promise - a distinction that turns out to be one of the deepest dividing lines in the new generation.

Hold those five, and one economic fact from my storage deep dive: on object storage, the format's real job is minimizing the number and maximizing the usefulness of ranged reads, because requests are the currency. Every design below is spending that currency differently.

Encodings, Explained Like You Are Human

Since the whole renaissance turns on encodings, let me build the intuition properly with tiny examples, because once these click, every format's architecture reads itself.

Start with dictionary encoding, the workhorse. A column of country names repeats endlessly: France, Japan, France, Brazil, Japan. Store the distinct values once in a dictionary - France is 0, Japan is 1, Brazil is 2 - and the column becomes 0, 1, 0, 2, 1, tiny integers instead of strings. Compression is enormous when cardinality is low, and - foreshadowing a key trick - some operations can run on the codes without ever rebuilding the strings.

Run-length encoding exploits consecutiveness: a sorted status column reading active, active, active, active, canceled, canceled becomes "active times 4, canceled times 2." Sorted and low-cardinality data collapses spectacularly.

Bit-packing notices that values fit in fewer bits than their type reserves: codes that never exceed 7 need 3 bits, not 32, so pack them shoulder to shoulder.

Frame-of-reference handles clustered numbers: order IDs 100,000,214 through 100,000,891 become "base 100,000,214" plus tiny offsets. Delta encoding does the same for sequences by storing differences - timestamps a second apart become a run of 1s, which then run-length encodes into almost nothing.

Encodings chain: delta, then run-length, then bit-packing, each feeding the next - and that chaining is what the research wave industrialized.

Two modern additions complete the toolkit. FSST brings the dictionary idea inside strings: it finds common substrings, builds a symbol table of fragments, and rewrites each string as symbol references, compressing well while keeping every individual string independently decodable - the property random access needs. ALP cracks floats by noticing that most real-world reals are decimals in disguise: 19.99 and 3.7 are integers scaled by powers of ten, so ALP finds the scaling per block, stores compact integers, and keeps exceptions exact, achieving what general codecs never could on the data type AI made ubiquitous.

Against all these stand the heavyweight codecs - Zstandard, Snappy, gzip - general-purpose compressors that treat bytes as bytes, find statistical redundancy anywhere, and pay for their generality in decode CPU and in opacity, since nothing can compute on their output without full decompression. The classic Parquet stack applies lightweight encodings first and a heavyweight codec over the top - belt and suspenders - with the heavyweight layer earning its cost when storage was slow and bytes were precious.

Now the research wave's discovery lands with full force: on modern fast storage, the heavyweight layer's decode cost often exceeds its transfer savings, while cascades of the lightweight encodings - chosen adaptively by sampling each chunk of data - match its compression and decode at memory speed, in SIMD-friendly patterns, sometimes without decoding at all. That single economic inversion - decode cost overtaking transfer cost - is the physics underneath every new format in this article. BtrBlocks proved it, FastLanes hardware-optimized it, ALP and FSST extended it to floats and strings, and Lance, Nimble, and Vortex are three different products of taking it seriously from day one.

Apache Parquet: The Incumbent, Renovating While Occupied

Architecture

The full treatment lives in my Parquet article, so the compressed version: row groups horizontally, column chunks within them, encoded and compressed pages within those, and a Thrift footer mapping everything with per-chunk statistics. Readers fetch the footer, prune row groups on statistics, and issue ranged reads for exactly the surviving columns' bytes. The design converts scans into a handful of large sequential reads, which is why it owns batch analytics on object storage.

Design Center

Scan-oriented batch analytics over structured data - compact at rest, prunable at plan time, readable by everything. That last property is the moat: thousands of independent implementations, exabytes written, and the guarantee that a Parquet file is a Parquet file everywhere.

Current State and Roadmap

The busiest era in its history, per my dedicated article: the variant type and shredding shipped for semi-structured data, geospatial types went native, format 2.13 released, and the two great campaigns run in public - the footer redesign (FlatBuffers versus a byte-offset index, attacking wide-table metadata costs) and the eighty-message versioning debate deciding how the format evolves without fragmenting. The AI-era additions queue behind them: a fixed-size list type for embeddings, the ALP float encoding imported from the research wave, and a contested File type for unstructured payloads.

Pros

Universality nothing else approaches, deep table-format integration - Iceberg, Delta, and Hudi are all Parquet-native - a compression and pruning story hardened by a decade at scale, and a community demonstrably willing to absorb its challengers' best ideas.

Cons

Random access is the structural weakness: retrieving one row means decoding a chunk of its row group, which multiplied across a billion point lookups is the gap the AI formats drove through. Footer costs bite at extreme width and extreme file counts - the renovation's whole motivation. Float-heavy and embedding-heavy data compresses and decodes below the modern frontier until the new encodings land. And evolution is deliberately slow - the price of a thousand implementations - which is precisely the opening the fast-moving newcomers exploit.

The Research Wave: The Ideas Underneath Everything New

Before the new formats, meet the ideas they are built from, because the renaissance's intellectual core is a handful of research results that changed what everyone believes about compression.

BtrBlocks, from the database group at TU Munich, made the foundational argument in 2023: for analytical data on fast storage, heavyweight general-purpose codecs are the wrong trade, and cascades of lightweight encodings - dictionary, run-length, frame-of-reference, delta, chained two or three deep and chosen per data sample - achieve comparable compression while decompressing at network speed, meaning decompression stops being the bottleneck even on multi-gigabit object storage links. The sampling-based, per-chunk automatic selection of encoding cascades is BtrBlocks' signature, and you will see it reappear in nearly every format below. As a format itself, BtrBlocks remains primarily a research artifact and reference implementation - CPU-oriented and enormously influential rather than widely deployed, the paper everyone builds on rather than the file everyone writes.

FastLanes, from CWI - the Amsterdam group behind much of columnar history - pushed further into hardware sympathy: a unified memory layout using virtual 1024-value vectors transposed for data-parallelism, so the same encoded bytes decode efficiently across any SIMD width and onto GPUs, plus expression encodings that chain codecs flexibly and multi-column compression that exploits correlations between columns - a frontier single-column designs cannot touch.

ALP - adaptive lossless floating point - cracked the float problem: reals compressed via adaptive decimal scaling far better and faster than general codecs manage. FSST did similarly for strings with random-access-friendly symbol tables.

The tell of the whole wave's success: ALP is now under evaluation inside Parquet itself, and the new formats below cite these papers the way engines cite Arrow.

The wave's collective lesson, worth one italicized sentence in your memory: modern columnar performance comes from many small, clever, chainable, hardware-native encodings chosen adaptively per data, not from one big codec applied uniformly. Every serious format now agrees. They differ on everything else.

Lance: The AI-Native Specialist

Architecture

Lance, from the team behind LanceDB, is the format that took the random-access problem personally. Its structural break with Parquet is the deletion of the row group: Lance 2.x organizes data so that any row is retrievable by position without decoding a neighborhood around it, using adaptive structural encodings that keep offsets navigable and pages independently fetchable. On top of the file layout sits what makes Lance a platform rather than just a format: a dataset layer with versioning, schema evolution, and - critically - secondary indexes as first-class citizens: vector indexes like IVF-PQ and HNSW for similarity search, scalar indexes for filtering, stored alongside the data they index. Blob-scale values - images, audio, documents - live natively next to structured columns, which is the multimodal story made physical.

Design Center

AI data, specifically the retrieval patterns AI creates: vector similarity search, filtered point lookups feeding models, random-access shuffles during training, multimodal datasets where the embedding, the metadata, and the source artifact belong together. Where Parquet asks "which million rows match," Lance asks "fetch me these ten thousand specific rows, now," and its claimed advantage on that pattern runs to two orders of magnitude.

Current State and Roadmap

Healthy and shipping: the 2.0 and 2.1 format generations delivered the structural-encoding architecture and better compression, the LanceDB ecosystem - embedded and serverful - gives it a native database, and adoption concentrates exactly where the design aims: vector search, feature retrieval, multimodal training corpora, with integration conversations reaching into the table-format world, including exploratory discussion of Lance as a file format within Iceberg-style tables. Roadmap direction: deeper index types, richer encoding adoption from the research wave, and the dataset layer maturing toward fuller lakehouse citizenship.

Pros

The best random-access and vector story in the field, genuine multimodal support, indexes as part of the format rather than an external system, and a coherent end-to-end stack for AI retrieval workloads.

Cons

Ecosystem breadth is the mirror image of Parquet's: one primary steward, one primary database, and general-engine support that is early - so choosing Lance today means choosing its stack. Scan-heavy classic analytics is not its game - Parquet remains better at the warehouse pattern. And the dataset layer's overlap with table formats creates an architectural either-or that enterprises will need to navigate.