The State of Apache Arrow in 2026: Ten Years In, the Invisible Standard Is Everywhere

The State of Apache Arrow in 2026: Ten Years In, the Invisible Standard Is Everywhere

In February 2026, Apache Arrow turned ten years old. The first commit landed on February 5th, 2016, and the anniversary passed the way Arrow itself operates: quietly, while running inside nearly every data tool you touched that day.

I have a house interest to declare here too, and in this case it is practically genetic. Arrow was co-created by people who founded and built Dremio. Jacques Nadeau, Dremio's co-founder, was Arrow's original PMC chair, and Dremio's founding engineers were among the earliest contributors, building the company's engine Arrow-native from day one, years before that was a fashionable phrase.

The project's origin braided together several threads: the Apache Drill community's in-memory format work, Wes McKinney's frustrations with pandas performance, and the Parquet community, where Julien Le Dem and other Parquet founders joined the early design sessions to build the in-memory complement to their on-disk format. Ten years later, that braid runs through pandas, Spark, Snowflake, DuckDB, Polars, and the AI stack.

So this article is my 2026 state of the project: what Arrow actually is beneath the buzzword, the full surface area it now covers, what the development pulse looks like this year, how deep the penetration really goes, the honest challenges, and why the next decade of Arrow is being written by AI workloads.

Arrow suffers from a strange fame problem: everyone has heard of it and few can explain it. Let me fix the second part.

What Arrow Actually Is: The Cost of Copies

Strip away everything else, and Arrow is an answer to one question: what should a table look like in memory? That sounds too small to matter. Here is why it is enormous.

Before Arrow, every system answered the question privately. Pandas had its internal layout, Spark had another, every database had its own, and every file format had its own. So whenever data crossed a boundary between systems - and analytics is nothing but data crossing boundaries - it had to be converted: serialized out of one representation, parsed into another, row by row, allocation by allocation.

Studies and painful experience suggested that big data systems were spending most of their CPU cycles not computing anything, just translating data between representations of itself. The industry was paying a permanent tax to disagreement.

Arrow's founding move was to make the in-memory representation a standard rather than an implementation detail. It specifies, byte for byte, how columnar data sits in memory: values of a column contiguous in fixed-width buffers, variable-length data like strings in value buffers with offset arrays pointing into them, null tracking in validity bitmaps, nested types built by composing these primitives. The layout is language-independent and machine-friendly, designed so modern CPUs can rip through it with vectorized instructions, and defined identically whether the process is C++, Java, Python, Rust, or Go.

The payoff is the elimination of translation. When two systems both speak Arrow, handing data between them is not a conversion - it is a pointer. The C Data Interface lets two libraries in the same process share Arrow data with zero copies, literally passing memory addresses. The IPC format lets processes and machines exchange Arrow data as a stream of buffers that the receiver uses as-is, no parsing step at all. The tax to disagreement drops to zero because the disagreement is gone.

The analogy I have used for years: before shipping containers, every transfer between ship, train, and truck meant unloading and repacking cargo by hand, and ports spent more effort repacking than moving. The container standardized the box, so cargo stopped being touched at boundaries. Arrow is the shipping container for tables. Nobody gets excited about the box. Everybody's goods move faster because the box is boring and identical everywhere.

One more foundation worth setting, since the two are eternally confused: Arrow and Parquet are complements, not competitors. Parquet is how a table sits on disk, optimized for compact storage and selective reading. Arrow is how a table sits in RAM, optimized for computation and exchange. Data typically lives as Parquet, gets read into Arrow, is computed on and shared as Arrow, and lands back as Parquet. The two communities have intertwined to the point that most official Parquet implementations, in C++, Rust, and Go, are literally developed inside Arrow repositories today. The founders designed them as two halves of one interoperability answer, and a decade later the halves have effectively merged their engineering.

A Decade in Six Chapters

Before surveying the present, a compressed history helps, because Arrow's current shape only makes sense as accumulation. Six chapters, roughly chronological.

Chapter one, 2016 to 2017, was the founding bet. The format specification, the first C++ and Java implementations, and the founding argument that a shared memory layout would pay for itself. Adoption was speculative, and the early integrations - pandas interchange, Spark's Python acceleration - were proofs of concept that the copy tax was real and removable.

Chapter two, 2018 to 2019, was language expansion. Rust, Go, JavaScript, and more joined, establishing the principle that Arrow's value scales with the square of its implementations, since every new language can now exchange with every existing one. Flight arrived at the end of this stretch, extending the standard from memory to the network.

Chapter three, 2020 to 2021, was the compute era. A 1.0 format with stability guarantees, the C++ compute kernels and what became Acero, Gandiva for expression compilation, and DataFusion growing inside the Rust repository. The project tested how far up the stack a standard should climb.

Chapter four, 2022 to 2023, was connectivity. Flight SQL matured, ADBC launched with its 1.0 specification, and nanoarrow appeared at the opposite extreme - Arrow as two embeddable C files. The project's center of gravity shifted from "represent data well" to "move data everywhere," which in hindsight was the decisive strategic turn.

Chapter five, 2024 to 2025, was restructuring and resilience. Language implementations moved to independent repositories, DataFusion graduated to its own Apache Top-Level Project, format additions like string views and run-end encoding landed for modern workloads, and the community absorbed the wind-down of its largest corporate patron without missing a release. The first Arrow Summit in Paris closed the chapter with the community meeting itself in person.

Chapter six is now: the anniversary, the AI reinterpretation, ADBC's adoption knee, and a contributor intake that skews toward Rust and toward the connectivity layers. Each chapter built on the last without discarding it, which is the quiet discipline that separates standards that endure from projects that pivot.

The Surface Area: Far More Than a Format

People who last checked on Arrow in 2019 think of it as a memory spec with a Python library. The 2026 project is a family of standards and libraries covering the full lifecycle of data in motion. A tour of the surface, layer by layer.

The Format Layer

The format layer remains the core: the columnar specification itself, now rich with a decade of carefully added types. Recent years brought:

• String view and binary view layouts, which store short strings inline and long strings by reference for dramatically faster string-heavy workloads
• List view variants
• Run-end encoding for compressed representation of repetitive data
• Smaller decimals down to 32 and 64 bits

Alongside the layout spec sit the interchange standards: the C Data Interface for zero-copy in-process sharing, the IPC streaming and file formats for crossing process and network boundaries, and canonical extension types so the ecosystem can agree on things like UUIDs, JSON, and geospatial data without forking the format. The discipline here is the story: format changes move slowly, through votes, because a hundred systems have to agree byte for byte, and the community has kept that discipline for ten years.

The Implementation Layer

The implementation layer is where the past two years reorganized the project. The main apache/arrow repository now holds the format specification and the C++ implementation with its Python, R, Ruby, and GLib bindings, while the other languages graduated to their own repositories:

• arrow-java
• arrow-rs for Rust
• arrow-go
• arrow-js
• arrow-dotnet
• arrow-swift
• arrow-nanoarrow

That split, mundane as it sounds, unclogged release cycles and let each language community move at its own pace, and the results show in the cadence: the main line shipped 23.0.0 in January 2026, a 23.0.1 security patch for the C++ IPC reader in February, and 24.0.0 in April, while Java shipped its own 19.0.0 in March with a proposal to raise the floor to JDK 17 for the next major, aligning with the broader Java modernization wave across the data stack.

Two implementations deserve their own sentences. Arrow-rs, the Rust implementation, has become an ecosystem force in its own right - the foundation under DataFusion, InfluxDB 3, and much of the new-generation lakehouse tooling in Rust - and in 2025 it attracted more first-time contributors than the main repository itself, 132 to 125. And nanoarrow, at the tiny end, ships the format as a pair of C files that any library can embed without adopting a heavyweight dependency, and its steady release train - 0.8.0 in February 2026 with string view building, LZ4 decompression, and broader packaging - has made "just embed Arrow" viable for R packages, database drivers, and anything else that wants the standard without the toolkit.

The Transport Layer

The transport layer is where Arrow stopped being only about memory. Arrow Flight is an RPC framework for moving Arrow data over the network at scale, skipping serialization on both ends. Flight SQL layers database semantics on top - queries, prepared statements, catalogs - so a database can expose one wire protocol that any Flight SQL client can consume, and Dremio, InfluxDB 3, and a growing set of engines ship it as a first-class interface. The long-running effort to build a Flight SQL ODBC driver, with sustained contribution from BigQuery-affiliated engineers among others, aims the same protocol at the vast installed base of ODBC tooling.

The Connectivity Layer

The connectivity layer is the newest and, I would argue, the most strategically important: ADBC, Arrow Database Connectivity, which deserves its own section.

The Compute Layer

The compute layer tells a graduation story. Arrow C++ ships Acero for query execution, and the Java lineage included Gandiva for expression compilation. But the headline is DataFusion, the Rust query engine that began life as an Arrow subproject and grew until it graduated into an independent Apache Top-Level Project, now the embedded engine inside a wave of commercial and open systems. Arrow's compute ambitions succeeded so thoroughly that they left home, which is the best possible outcome for a foundational project: the standard stays neutral, and the engines built on it compete above it.

ADBC: Fixing the Last Slow Mile

If I had to pick the single most consequential Arrow initiative of this decade so far, it is ADBC, because it attacks the last place where the copy tax survived untouched: the database driver.

The absurdity it fixes is easy to state. We spent fifteen years building columnar databases, columnar file formats, and columnar memory, and then connected them to applications through JDBC and ODBC - APIs designed in the early 1990s that hand data over row by row. A columnar database executing a columnar query for a columnar client would pivot results into rows at the driver boundary so the client could pivot them straight back into columns. Every analytical tool in the world was paying this toll on every query, and it was invisible only because it was universal.

ADBC is the columnar replacement: a vendor-neutral API where applications ask for data and receive Arrow, full stop. Drivers for Arrow-native systems pass data through essentially untouched. Drivers for row-oriented systems do the conversion once, inside the driver, where it can be optimized, instead of in every application. It deliberately complements rather than replaces Flight SQL: ADBC is the client-side API, Flight SQL is a wire protocol a server can speak, and an ADBC driver can use Flight SQL, a native protocol, or anything else underneath.

The 2026 status is a project hitting its adoption knee. The libraries shipped version 23 in April, with the API specification itself at 1.1 and a 1.2 milestone underway focused on richer metadata and catalog capabilities. The driver roster now covers Snowflake, BigQuery, DuckDB, PostgreSQL, SQLite, Flight SQL generally, and a newly contributed Go driver for Databricks, with a startup called Columnar, founded around core Arrow contributors including Ian Cook, launching commercial ADBC drivers for Redshift, MySQL, SQL Server, and Trino to fill the long tail.

Adoption stories carry real numbers: DuckDB reported query time reductions beyond 90 percent in many applications versus the old driver path, Microsoft adopted ADBC for Power BI connectivity, and the driver manager work now spans Python, Java through JNI bindings to native drivers, and as of this spring, Node.js on NPM. A decade from now, I suspect we will look at row-oriented drivers for analytics the way we now look at hand-rolled CSV parsers: a thing everyone did, and nobody could quite explain why.

The Development Pulse: A Community That Outgrew Its Patron

The health of a ten-year-old open source project is a fair question, and 2025 gave Arrow a stress test with an unusually clean answer. Voltron Data, the company that had employed a large concentration of Arrow maintainers and hosted project infrastructure, wound down operations in 2025. A project overly dependent on one patron dies or drifts when this happens, and the fact that Arrow's year-end community report treats the event mostly as...