Block vs. Object Storage: A Deep Dive Into the Foundation of Modern Data, and How the Lakehouse Made the Slow Option Fast

Block vs. Object Storage: A Deep Dive Into the Foundation of Modern Data, and How the Lakehouse Made the Slow Option Fast

Here is one of the strangest and most consequential plot twists in the history of data infrastructure: over the past decade, the analytics industry deliberately moved its data onto the slowest storage it could find, and got faster.

Amazon S3 answers a read request in tens to over a hundred milliseconds. A modern block volume answers in one or two, and local NVMe in microseconds. By the only metric storage vendors printed on the box, object storage was a fifty-to-hundred-fold step backward.

Yet today the world's analytical data, the lakehouses, the training sets, the event histories, lives overwhelmingly on object storage, queries against it come back in seconds or less, and nobody serious is moving back.

That plot twist is not a paradox. It is a story about which properties of storage actually matter at scale, and about a decade of brilliant engineering, in file formats, table formats, and query engines, that systematically neutralized every weakness the latency number represented.

This article is the full deep dive. What block and object storage actually are, mechanically, without hand-waving. The honest comparison across latency, throughput, scalability, availability, durability, cost, and consistency, with real numbers. The specific wrinkles object storage inflicts on data workloads, and then the heart of the piece: how Apache Parquet, Apache Iceberg, and engines like Dremio overcome each wrinkle, layer by layer, turning eleven nines of cheap durability into interactive analytics.

By the end, the plot twist should feel inevitable, and you should be able to reason about any storage decision, and any vendor's storage claim, from first principles.

Block Storage, Mechanically: The Disk Abstraction

Start with the elder statesman, because everything else is defined against it. Block storage presents the oldest abstraction in computing: a device that stores fixed-size blocks of bytes, addressable by number, readable and writable in place. It is the disk, virtualized.

A block volume knows nothing about files, folders, or meaning. It knows that block 4,000,017 contains these 4,096 bytes, and it will read or overwrite them on demand, fast. Everything humans recognize as files comes from the filesystem - ext4, XFS, NTFS - that the operating system layers on top, translating names and directories into block addresses.

In the cloud, block storage means network-attached virtual disks: AWS EBS, Azure Managed Disks, Google Persistent Disk. A volume attaches to one server, the server formats and mounts it, and applications get the full POSIX contract: open, seek, read, write, append, modify in place, all at low latency. Top-tier volumes like EBS io2 Block Express deliver latencies under one to two milliseconds with hundreds of thousands of IOPS, and local NVMe instance storage goes lower still, into microseconds, at the price of vanishing when the instance does.

The strengths follow from the abstraction. Latency is the headline: sub-millisecond random reads and writes are what databases are built on, which is why every serious transactional system - Postgres, MySQL, SQL Server, SAP HANA with its sub-millisecond hot-tier requirements - runs its data files and write-ahead logs on block storage, no exceptions. In-place modification is native: change one record, rewrite one block. And the semantics are the ones fifty years of software assumes.

The limits are just as structural. A volume attaches to one machine, so sharing requires a database or a network filesystem on top. Capacity scales up, not out: volumes have size ceilings measured in tens of terabytes, and growing past them means sharding across volumes, which is your problem. Durability is good but engineered per volume, with annual failure rates that make backups mandatory rather than paranoid. And the cost per gigabyte runs several times object storage, because you are paying for performance-grade media and reserved capability whether you use it or not.

Block storage is a sports car: unmatched at what it does, expensive to park, and the wrong vehicle for hauling freight.

A brief word on the middle sibling, file storage - NFS, SMB, cloud services like EFS: a real filesystem, mountable by many machines at once, with latency and cost between the other two. It earns its place for shared application directories and legacy workloads that demand POSIX across a fleet. For the analytical story this article tells, it plays a supporting role at most, and the main event is the other extreme.

Object Storage, Mechanically: The Warehouse of Sealed Boxes

Object storage threw away the disk abstraction entirely and asked a different question: what is the simplest possible contract for storing unlimited stuff durably?

The answer: a flat keyspace of immutable blobs behind an HTTP API. An object is a key - a string that may look like a path but is just a name - a value, the bytes, up to five terabytes, and metadata. The operations are few and blunt: PUT an object, GET an object or a byte range of it, DELETE it, LIST keys by prefix. There are no directories, no seek-and-modify, no append. To change one byte of an object, you upload a replacement object.

That is the whole contract, and its bluntness is the point, because every simplification buys scale.

Behind the API sits the engineering that made S3 and its kin - Azure Blob, Google Cloud Storage, and the S3-compatible world of MinIO, Ceph, R2, Wasabi and company - the largest storage systems ever built. Each object is erasure-coded: split into shards with redundancy math - Reed-Solomon coding - spread across many drives, racks, and availability zones, so that multiple simultaneous hardware failures lose nothing. That is where the famous durability figure comes from: eleven nines, 99.999999999 percent annual durability, a number so extreme it means expected loss of one object per hundred billion per year.

Availability is engineered separately and regionally, typically 99.9 to 99.99 percent for standard tiers. And the scale numbers are the ones worth reading twice: S3 alone holds over 500 trillion objects and serves peaks beyond 200 million requests per second. There is no volume to fill, no cluster to grow. The keyspace is effectively infinite, and throughput scales horizontally: one client reading one object gets modest speed, but a thousand clients reading a thousand objects, or a thousand ranges of one huge dataset, aggregate to staggering bandwidth.

Cost lands at a fraction of block storage per gigabyte, with tiering to archive classes for a further order of magnitude, though the pricing model has teeth elsewhere: every request is billed, and egress - moving data out to the internet or across clouds - is the line item that has ruined many a budget.

One more property graduated from caveat to strength in recent years, and it matters enormously for what comes later. Object stores were historically eventually consistent: a read after a write might see stale data, which forced analytics tools into elaborate workarounds. On December 1, 2020, S3 shipped strong read-after-write consistency for all operations at no cost, and the serious S3-compatible stores followed. More recently, conditional writes arrived: PUT operations that succeed only if the key does not exist, or only if it still carries an expected version, giving object storage its first honest compare-and-swap primitive.

File this fact away. It is the hinge of the transactional story later in this article.

So the trade emerges in full: object storage exchanged latency and mutability for effectively infinite scale, extreme durability, horizontal throughput, and radical cheapness. Fifty to a hundred fifty milliseconds to first byte, versus one or two. No append, no in-place edit, no real directories. Request bills and prefix-level rate limits - on the order of thousands of writes and several thousand reads per second per key prefix - that punish designs treating it like a filesystem.

It is a warehouse of sealed boxes: you can store more boxes than you could ever count, retrieve any of them from anywhere, and never worry about losing one, but you cannot edit a box's contents on the shelf, and walking to the warehouse takes longer than reaching into your desk drawer.

The Scorecard, Side by Side

Compressing the comparison into the dimensions that decide real architectures:

• Latency: block wins by two orders of magnitude - sub-two milliseconds against fifty to a hundred fifty - with the newer premium object tiers like S3 Express One Zone splitting the difference at single-digit milliseconds for a price and a single-zone durability trade.
• Throughput: object wins at scale, because aggregate parallel bandwidth across a distributed fleet beats any single volume's pipe, and the win grows with parallelism.
• Scalability: object, without contest - infinite keyspace and horizontal everything against per-volume ceilings and single-attach.
• Durability: object, eleven nines by erasure-coded design against per-volume failure rates that mandate backup regimes.
• Availability: comparable when architected properly, with object's multi-zone replication built in and block's requiring you to build it.
• Cost per gigabyte: object by several fold, and by an order of magnitude with tiering, while block charges premium rates and object claws some back through request and egress billing.
• Semantics: block wins completely - full filesystem behavior against PUT-GET-LIST-DELETE and no in-place anything.
• Sharing and access: object, HTTP from anywhere with IAM-grade policy, against one-machine attachment.

Read the scorecard and the classical division of labor writes itself, and it remains correct: transactional databases, low-latency serving, boot volumes, and scratch space live on block. Archives, media, backups, and data lakes live on object. AI stacks split the same way instinctively: training corpora on object for scale and throughput, the training runs' working state on block and NVMe for latency, model artifacts back to object for serving and history.

The interesting question, the one the rest of this article answers, is the workload that refused the division: interactive analytics. It wants the scale, durability, and cost of object storage, its datasets are petabytes, and it wants answers in seconds, which the latency column says object storage cannot give.

The lakehouse is the decade-long engineering project that proved the latency column wrong, or more precisely, made it irrelevant. Here is how, layer by layer.

How Analytics Ended Up Here: A Short History of the Migration

Before dissecting the wrinkles, the migration itself deserves a page, because knowing why analytics moved onto object storage explains why nobody is moving back, latency numbers notwithstanding.

The story starts with Hadoop's HDFS, the first system to make a deep philosophical bet that data should live on cheap commodity storage with the intelligence layered above. HDFS got the economics right and the operations wrong: clusters coupled storage and compute on the same machines, so growing one meant growing both, capacity planning was a permanent argument, and keeping thousands of disks healthy was a career.

When the cloud arrived, it offered a clean divorce: object storage as a bottomless, maintenance-free data tier, compute as an elastic fleet rented by the minute, connected by a network fat enough to make the separation workable. That divorce - storage-compute separation - is the single most important architectural shift of the era, and its logic is worth stating crisply.

Coupled systems size for peak: your cluster must be big enough for the Monday morning rush and the month-end close, and it idles at that size all week. Separated systems size independently: storage grows with data, smoothly and cheaply, while compute scales to zero between workloads and to enormous during them. Different teams can bring different engines to the same bytes without copying, which is the multi-engine lakehouse in embryo. And durability becomes the provider's problem, engineered once at eleven nines, instead of every cluster's ops team's problem, engineered repeatedly at whatever nine the budget allowed.

The warehouses proved the model first, Snowflake's architecture being the famous demonstration that separation plus caching could beat coupled systems at their own performance game, and the open world followed with the harder version: separation without a proprietary engine in the middle. That required the storage tier's rawness to be tamed by open layers anyone could implement, which is precisely the Parquet-Iceberg-engine stack this article walks.

By the time the AI era arrived demanding petabyte corpora with fleet-wide parallel access, the question was settled: the economics and elasticity of the separated model were worth any engineering it took to hide the substrate's rough edges.

The rest of this article is that engineering.

The Wrinkles: What Object Storage Does to Data Workloads

First, name the enemies precisely, because each one gets its own counter. Running analytics naively on object storage hurts in six specific ways:

1. Per-request latency. Every GET costs tens of milliseconds before the first byte, so a query whose plan touches ten thousand objects serially spends its life waiting on round trips.
2. Listing is slow and expensive. Discovering what files exist by LIST calls over big prefixes takes seconds to minutes and bills per request, and the old Hive-style pattern of listing directories to plan queries dies at scale on this alone.
3. No atomic multi-file operations. A data update that writes fifty files has no way to make them appear simultaneously, so readers can see half-finished states, and two writers can silently trample each other.
4. No append or in-place edit. Immutable objects mean every modification pattern that databases take for granted - append a row, update a record - must be reinvented as write-new-plus-supersede.
5. Small files are poison. Millions of tiny objects