The State of Apache Parquet in 2026: The Quiet Format Enters Its Loudest Decade

The State of Apache Parquet in 2026: The Quiet Format Enters Its Loudest Decade

Apache Parquet is thirteen years old, holds more of the world's analytical data than any other format, and, for most of its life, has been the least dramatic project in the data stack. It sat at the bottom, it worked, and the interesting arguments happened in the layers above it.

That era is over. As someone who reads the Parquet dev mailing list every week for my newsletter, I can report that 2026 is the busiest, most consequential stretch of Parquet development in a decade. The past year alone brought a native variant type for semi-structured data, first-class geospatial types, a new format release, an active redesign effort for the file footer, proposed types for embeddings and unstructured blobs, a new floating-point encoding, and the single longest discussion thread I have seen on that list: eighty-plus messages debating nothing less than the future of Parquet versioning itself.

Why is the quiet format suddenly loud? Two forces converged. The lakehouse era made Parquet the shared substrate under every table format, so every ambition of Iceberg and Delta eventually becomes a demand on Parquet. And the AI era arrived with workloads, embeddings, semi-structured context, wide feature tables, unstructured payloads, that the format's 2013 assumptions never anticipated. Parquet is being renovated while fully occupied, which is the hardest kind of engineering and the most interesting kind to watch.

This article is my 2026 state of the project: a proper refresher on how Parquet works, what shipped this year, what the dev list is fighting about, how the ecosystem of implementations has reorganized, and what it all means for the people building on top. As always, the goal is that the logic clicks, so the next Parquet headline you see explains itself.

Thirteen Years in Four Chapters

A compressed history sets the stakes for the present, because Parquet's current renovation only reads correctly against what came before.

Chapter one, 2013 to 2015, was the founding. Engineers at Twitter and Cloudera, with Julien Le Dem among the creators, built a columnar file format for the Hadoop ecosystem, drawing on the record-shredding ideas from Google's Dremel paper to handle nested data properly, something earlier columnar attempts had fumbled. The bet was that analytical storage should be columnar, compressed, and self-describing, and that an open format would beat every vendor's private one. The bet was not obviously right at the time. Row-oriented formats dominated, and columnar was a warehouse-vendor specialty.

Chapter two, 2016 to 2019, was the victory. Spark made Parquet its default, every SQL-on-Hadoop engine standardized on it, cloud object storage made file formats matter more than databases, and the Arrow project arrived as the in-memory complement, with the two communities intertwining from the start. By the end of the chapter, Parquet was less a choice than an assumption, and the exabytes began accumulating.

Chapter three, 2020 to 2024, was the lakehouse consolidation. Iceberg, Delta, and Hudi all chose Parquet as their substrate, which quietly changed the format's job description: it was no longer just a file people read, it was the physical layer of a transactional table abstraction, and every table-format ambition - statistics for planning, encryption for governance, column-level everything - arrived as a requirement on Parquet. Development in this chapter was steady and unglamorous: page indexes, bloom filters, modular encryption, better encodings, the compounding infrastructure work that made the layers above possible.

Chapter four is now, and its character should be clear from everything above: the AI era arrived with new shapes of data and new intensities of access, the lakehouse's next generation is negotiating with the format layer in real time, and a community that spent a decade in maintenance mode is running its most ambitious renovation, in public, with the versioning question as the constitutional debate that will govern how all the rest lands. Thirteen years in, the format's second act is genuinely more interesting than its first.

A Proper Refresher: How Parquet Actually Works

Everything in this article depends on the file anatomy, so let me build the mental model quickly and honestly, because most Parquet explanations stop one level too shallow.

A Parquet file organizes a table in three nested layers. The file is divided horizontally into row groups, each holding some slice of the rows, commonly in the hundreds of thousands. Within each row group, data is organized vertically into column chunks, one per column, so all the values of one column in that row slice sit together. Within each column chunk, values are stored in pages, the smallest unit of encoding and compression, typically around a megabyte.

This nesting is the whole trick. Row groups let engines parallelize and skip horizontally. Column chunks let queries read only the columns they touch. Pages let the format apply the right encoding per batch of values: dictionary encoding when values repeat, run-length encoding when they repeat consecutively, bit packing for small integers, delta encodings for sorted data, with general compression like Zstandard layered on top. The encodings are why Parquet files routinely land at a fraction of the size of the same data as CSV or JSON, and the layout is why queries can ignore most of a file's bytes.

Then comes the part this year's arguments revolve around: the footer. At the end of every Parquet file sits a metadata block, serialized with Apache Thrift, describing everything a reader needs: the schema, the location of every row group and column chunk, and statistics - minimum and maximum values, null counts - per column per row group. Readers open a Parquet file by reading the footer first, and the statistics power the pruning that makes analytics fast: a filter on date can skip every row group whose date range cannot match, before decompressing a single page.

Auxiliary structures extend the same idea: page indexes push min and max tracking down to page granularity, and bloom filters answer "is this value definitely absent" for high-cardinality columns.

Hold two design facts from this tour, because the rest of the article pulls on them. First, the footer must be read before anything else, and it must be substantially decoded even when a query wants one column of a thousand - a property of the Thrift serialization that was harmless when tables were narrow and files were opened rarely. Second, the format's power comes from types and statistics: every capability Parquet gains arrives as a new logical type, a new encoding, or a new statistic, which is exactly the shape of everything that shipped this year.

What Shipped: Variant, Geospatial, and Format 2.13

Start with the ratified and released, because 2026 opened with Parquet formally announcing two of the biggest type-system additions in its history.

The variant type went official

In February 2026, the Parquet community announced native support for the Variant type, the binary encoding and shredding specifications for semi-structured data that I have written about at length in the Iceberg context, because Iceberg v3's variant support is built directly on these Parquet specs. The design lives at the Parquet layer on purpose: the binary encoding replaces JSON text with a compact, offset-navigable representation, and shredding extracts frequently occurring fields into real Parquet columns with real statistics, with engines able to supply an explicit shredding schema when read patterns are known or let inference decide.

Because the specification is Parquet's, every engine and table format that implements it shares one physical representation, which is why Spark can write shredded variants that Dremio reads transparently. The dev list traffic since the announcement shows a spec in the hardening phase: threads on realistic variant depth limits, on how pre-variant readers should behave when they meet variant columns, and on where shared components like the JSON parser should live. This is what success looks like for a format feature: the arguments move from "should it exist" to "what happens at the edges."

Geospatial became a first-class citizen

Also in February, Parquet announced native geometry and geography logical types. For years, spatial data lived awkwardly outside mainstream analytics, in specialized formats or as opaque well-known-binary blobs that columnar engines could store but not understand, with the GeoParquet community convention bridging the gap admirably from the outside. The native types bring coordinates, spatial reference systems, and geometry semantics into the format itself, with statistics such as bounding boxes enabling spatial pruning the same way min and max enable numeric pruning.

The follow-on dev list work has the same healthy hardening shape: clarifying coordinate reference system string formats, aligning with the parallel geospatial work in Iceberg v3, which adopted geometry and geography types in the same wave. Logistics, climate, mobility, and location intelligence workloads just got a columnar home that the whole ecosystem shares.

And the release train delivered

Parquet format 2.13.0 went through its release process this spring, carrying the accumulating spec work, while parquet-java shipped its 1.17 line with a 1.17.1 patch following. The community also passed a set of small, telling votes: defining ordering for the legacy INT96 timestamps, adopting IEEE 754 total ordering with NaN counts so floating point statistics finally handle NaN values coherently, and making a redundant schema-path field optional to shave footer weight. Individually minor, collectively these are the format sanding down decade-old ambiguities, the kind of work that only happens when implementers compare notes at scale.

The Eighty-Message Thread: The Future of Parquet Versioning

Now the argument that towers over the season's dev list: what does a Parquet version even mean, and how should the format evolve from here?

The problem is one Parquet earned through success. The format nominally has versions, and files carry a version marker, but the marker long ago stopped describing reality. Features landed in the specification one by one over a decade, implementations adopted them at wildly different speeds, and "Parquet 2" ended up meaning different things to different writers. The practical result is that nobody negotiates compatibility by version number. Engines make conservative feature-by-feature choices about what to write, defaulting to the lowest common denominator because they cannot know what readers will meet their files. That conservatism has a real cost: excellent features like delta encodings and modern statistics sit underused for years because writers dare not emit what some reader somewhere might choke on.

The thread, running past eighty messages with contributors from across the implementer ecosystem, is wrestling with the way out, and the option space is instructive.

• One direction formalizes feature flags: files declare exactly which capabilities they use, readers declare what they support, and compatibility becomes a checklist rather than a version comparison - an approach with clear precedent in how table formats above Parquet handle the same problem.
• Another direction argues for meaningful version milestones, a genuine "Parquet 3" that bundles the modern feature set - footer improvements, new types, better defaults - into a named target that the ecosystem can rally around and test against, with the marketing clarity that a decade of accumulated features has lacked.

The companion thread on documenting which features belong to which versions shows the community doing the archaeology either path requires. I will not predict the outcome, but I will name what is actually at stake, because it is bigger than labeling. The versioning decision determines Parquet's metabolism: how fast the format can absorb the AI-era additions discussed below without fracturing into dialects. A format read by thousands of independent implementations has one asset above all others - the guarantee that a Parquet file is a Parquet file - and the versioning thread is the community redesigning how to grow without spending that asset. It is the most important boring argument in the data stack right now.

The Footer Problem: Renovating the Front Door

The second great campaign of 2026 attacks the footer, and this one comes with a working group, regular sessions, and competing proposals.

The complaint, precisely stated: the Thrift-serialized footer must be parsed monolithically. A reader wanting the schema and the location of three columns must decode metadata for all one thousand columns, because Thrift's compact protocol does not support jumping selectively into the structure. When tables were dozens of columns and files were opened once per long scan, nobody noticed. Then came the modern workloads: feature tables thousands of columns wide, machine learning pipelines opening thousands of files per second, interactive engines where footer decode time is visible in query latency, and metadata-heavy features - page indexes, bloom filters, variant shredding statistics - all growing the footer they attach to. Measurements across the ecosystem put footer decoding at a startling share of some scan workloads, and the wide-table AI cases suffer worst.

Two remedial philosophies are on the table, and the contrast is a beautiful engineering study.

The first replaces Thrift with FlatBuffers, a serialization format designed for zero-copy access: readers map the footer bytes and jump directly to the pieces they need, decoding nothing they do not touch. It is the thorough fix, and it is also a breaking change to the most compatibility-critical bytes in the analytics world, which is why it has been debated carefully for over a year.

The second philosophy, advanced this spring as an alternative, keeps Thrift but adds a ligh