File Encryption for the Lakehouse: The Terminology, the Machinery, and the Hard Problem of Interoperable Encrypted Tables

Terminology Bootcamp

Encryption conversations run on a vocabulary that gates comprehension, so here is the working glossary, built in dependency order, each term earning the next.

Plaintext and ciphertext are the before and after: readable data, and the output of encryption, which should be computationally indistinguishable from random bytes. That indistinguishability, incidentally, is why compression should happen before encrypting: ciphertext has no redundancy left to compress.

Symmetric encryption uses one key for both directions, and it is what bulk data encryption always uses, because it is fast. The universal standard is AES (Advanced Encryption Standard), hardware-accelerated on essentially every modern CPU through dedicated instructions, which is why encrypting terabytes is computationally cheap in 2026. Asymmetric encryption (the public-and-private key kind) is too slow for bulk data and appears in this story only around the edges: wrapping keys and authenticating parties.

Modes determine how AES, which natively scrambles single 16-byte blocks, extends to real data. One distinction here does enormous work. AES-CTR (counter mode) encrypts efficiently and provides confidentiality only: an attacker cannot read the data but can flip bits and splice sections without detection. AES-GCM (Galois/Counter Mode) provides authenticated encryption: confidentiality plus an integrity tag, so any tampering, truncation, or splicing is detected at decryption. Modern designs default to GCM, and when you see a format offer a CTR variant, it is a deliberate performance-versus-integrity trade for specific situations.

Authenticated modes require a nonce or initialization vector (IV), a never-reused-per-key value whose correct handling is one of those details specifications exist to get right so you cannot get it wrong. AAD (additional authenticated data) extends GCM's integrity beyond the ciphertext: extra context (a filename, a module identifier) that is not encrypted but is bound into the integrity tag, so ciphertext moved to a different context fails to decrypt. Hold this one - it is the elegant trick that stops an attacker from swapping encrypted pieces between files.

Envelope encryption is the architecture everything at scale uses. Encrypting a petabyte directly with one master key is operationally insane, so instead: each file gets its own DEK (data encryption key), generated randomly at write time. The DEK encrypts the data, and then the DEK itself is encrypted (wrapped) by a higher key and stored alongside the data it protects. The wrapping key may itself be wrapped by another, yielding hierarchies: DEKs wrapped by KEKs (key encryption keys), wrapped by a master key. The master key lives in a KMS (key management service), a hardened system, often backed by an HSM (hardware security module), that never releases the master key at all: clients send wrapped keys to the KMS and receive unwrapped ones, every operation authenticated, authorized, and audited.

The beauty of the envelope: bulk data never moves for key operations; rotating or revoking a master key means re-wrapping small keys, not re-encrypting petabytes; and the KMS audit log becomes the ledger of who could read what, when.

Key rotation is the practice of retiring keys on schedule, limiting how much any single compromised key exposes. Crypto-shredding is rotation's dramatic cousin: destroy a key, and everything encrypted under it becomes permanently unreadable, which converts data deletion (nearly impossible to prove across replicated immutable storage) into key deletion (instant and provable) - a property privacy regulation made valuable beyond measure.

Finally, the client-side versus server-side axis: SSE with provider-managed keys, SSE with your KMS keys (which adds your audit and revocation but still decrypts for any bucket-authorized caller), SSE with customer-provided keys, and full client-side encryption (where the storage never sees plaintext). The lakehouse machinery below is the client-side end of that menu, made multi-engine.

Why Bucket Encryption Was Never Enough

Start with the question that stalls half the encryption conversations I have: our object storage is already encrypted, so what problem remains?

Server-side encryption (the SSE in your S3 configuration) means the storage service encrypts bytes before writing them to its disks and decrypts them on every authorized read. It is genuinely valuable and genuinely narrow: it protects against threats to the physical storage layer - stolen drives, decommissioned hardware, a breach beneath the service's API. Against everything above that line it does nothing, because the service transparently decrypts for any caller with bucket permissions. A leaked credential, an over-broad IAM role, a compromised service, a malicious insider with storage access - every one of them reads plaintext, because to the storage API, they are authorized.

Threat modeling makes the gap precise. Server-side encryption answers "what if someone steals the disks." It does not answer "what if someone gets into the bucket" (which is the overwhelmingly more common incident), nor "what if the storage provider itself must be outside the trust boundary" (the sovereignty and regulated-industry requirement), nor "how do I prove to an auditor that a specific column of personal data was unreadable to everyone without a specific key." Those questions require the data to be encrypted before it reaches storage, under keys the storage service never holds, decryptable only by clients you control. That is client-side encryption, and in the lakehouse, where the clients are a fleet of heterogeneous query engines sharing files, client-side encryption is exactly the interoperability puzzle this article exists to work through.

The honest framing, which I will repeat at the end: bucket-level encryption plus access control is a legitimate, sufficient posture for plenty of estates. The machinery below is for the estates where it is not - regulated data, multi-tenant platforms, sovereignty constraints, defense in depth mandates - and the population of those estates grows every year the AI era pushes more sensitive data into analytical reach.

The Layers: Where You Can Encrypt, and What Each Buys

With the vocabulary loaded, the design space becomes a clean question of layers, each protecting against more and costing more.

• Layer one, transport: TLS on every connection. Table stakes, universally deployed, protects data in motion, and says nothing about data at rest.
• Layer two, storage-service encryption: the SSE family. Protects the physical layer, satisfies the baseline checkbox, transparent to everything above, and powerless against credentialed access.
• Layer three, whole-file client-side encryption: encrypt each object before upload as an opaque blob. Maximum confidentiality and the death of analytics: an encrypted blob has no readable footer, no statistics, no ranged reads, so every query downloads and decrypts entire files. This layer is for archives and backups, not tables, and its failure at analytics is precisely what motivated the next layer.
• Layer four, format-aware encryption: encryption designed into the file format itself, so that the columnar machinery (footers, statistics, selective column reads, pruning) survives. This is Parquet Modular Encryption's layer, and it is where the lakehouse story lives, because it is the only layer that delivers client-side protection and analytical performance simultaneously.
• Layer five, field-level and application encryption: individual values encrypted before they ever enter the data platform, by the producing application. Strongest isolation, and the values become opaque to the platform (no filtering, no aggregation, no statistics on those fields), so it suits the narrow tier of ultra-sensitive identifiers (often paired with tokenization) rather than general columns.

The pattern to internalize: each layer up the stack shrinks the set of parties who can see plaintext, and shrinks what the platform can do with the data. Format-aware encryption exists because it bends that trade better than any other point, keeping plaintext away from storage and network while preserving nearly everything analytics needs. Defense in depth means running several layers at once - TLS plus SSE plus format-level for the sensitive tables - and the design work is choosing where each table's requirements land.

Parquet Modular Encryption: The Format-Aware Foundation

Parquet Modular Encryption, developed in the Parquet community with Gidon Gershinsky as its long-time driving force, is the piece that made layer four real, and its design rewards a close look because every property was chosen to preserve exactly what makes Parquet valuable.

The core move: encrypt Parquet's modules, not its file. Each unit of the format - data pages, dictionary pages, footer, indexes - is encrypted independently with AES (GCM by default), after encoding and compression have done their work (order matters, per the compression article). Because modules encrypt independently, the read path survives intact: a reader fetches and decrypts the footer, plans as always, and then fetches and decrypts only the pages the query touches. Selective column reads, predicate pushdown, ranged GETs - the whole economic model of analytical storage - all preserved under encryption. That single property is the difference between encryption you can afford on analytical tables and encryption you cannot.

The key model is columnar, and this is where governance enters the format: different columns can be encrypted under different keys. The salary column under one key, the email column under another, the non-sensitive columns under a footer key or left plaintext. A reader possessing only some keys can read exactly those columns, and fine-grained access control acquires a cryptographic enforcement layer beneath the policy layer: even a reader who bypasses every engine and opens the raw file gets only the columns whose keys it holds.

The footer gets special treatment because it is special: it holds the schema, the offsets, and the statistics - and statistics leak (min and max values of an encrypted column are data). Encrypted-footer mode, marked by the PARE magic bytes replacing Parquet's usual signature, encrypts the whole footer under its own key, hiding schema and statistics from keyless readers, while a plaintext-footer variant keeps legacy readers able to see the file's structure and the unencrypted columns, trading some leakage for compatibility.

Integrity runs through everything via AAD: each module's encryption binds identifiers of its position (which file, which column, which page) into its authentication tag, so an attacker with storage access cannot splice pages between files, swap one file's column chunk into another, or roll a column back to an older version without decryption failing loudly. Tamper-proofing, not just secrecy.

What the format deliberately does not define is where keys come from: it specifies key metadata fields and leaves key management to the layer above - a modularity that seemed like a gap and turned out to be foresight, because the layer above now exists.

Iceberg Table Encryption: The Envelope Around Everything

Parquet encrypts files. Tables are more than files: they are metadata trees, manifests full of statistics, paths, and structure, and an encrypted table whose metadata is plaintext leaks its shape, its stats, and its history.

Iceberg 1.11, released May 19, 2026, closed that gap with table-level encryption - the feature I flagged as a design discussion in my Polaris coverage, now shipped - and its architecture is the envelope pattern executed across a whole table format.

The key hierarchy has three tiers, each earning its place. At the top, a table master key, living in your KMS.