High-Bandwidth Flash offers efficient storage for model weights

Comments

Large Language Models (LLMs) demand immense amounts of memory, and the more people use them, the more memory is required. Memory makers responded by accelerating plans to build new memory fabs, with a focus on high-bandwidth memory (HBM) and DRAM, the first of which is scheduled to start production in 2027.

But the demand for memory may also provide an opportunity for new ideas to find footing. One of these is a tricked-out version of the kind of memory that lives in an SD card or a thumb drive-High Bandwidth Flash (HBF). It essentially takes the ideas that made HBM successful-stacking multiple chips to increase capacity and bandwidth-and applies them to the NAND flash memory commonly used for data storage in SD cards, thumb drives, and smartphones, among many other devices.

"People ask, 'How in the world does this make a grain of sense? Flash is enormously slow,'" says Jim Handy, general director at semiconductor market research firm Objective Analysis. He explains that while NAND flash is generally lacking in bandwidth, HBF will help alleviate that concern. "[Flash] is atrociously slow for writes, but for reads, it can be coaxed to go pretty fast. And High Bandwidth Flash is going to be coaxed to do that."

What is High Bandwidth Flash?

NAND flash stores data as a trapped electric charge in arrays of floating-gate transistors, organized into blocks and pages rather than individually addressable bytes. It's non-volatile, too, which means data persists without power. These traits make flash a good choice for long-term storage. It can store more bytes in the same area than DRAM, and it doesn't require power-hungry capacitors that need constant refreshing to hold their charge.

But the mechanisms that make flash dense and non-volatile also make it slow to write to, as pushing charge into and out of an insulated gate takes longer than charging a capacitor. The latest flash interface standard can support memory bandwidth up to 4.8GB/s per die. That's not bad for many situations, and NAND is widely used in high-performance long-term storage, such as solid state drives.

However, DDR5 provides bandwidth up to 70.4GB/s per DIMM (excluding overclocked memory), and HBM4E can reach up to 3.6TB/s per stack-a roughly 750-fold bandwidth advantage for HBM4E over flash.

Hoshik Kim, senior vice president of memory systems research at SK Hynix, says HBF improves bandwidth with packaging techniques similar to HBM. "By applying advanced 3D packaging and vertical stacking techniques to NAND flash, HBF can deliver vastly higher bandwidth than standard NVMe storage," he says.

Much as HBM stacks DRAM, HBF stacks NAND flash dies to create a memory-dense chip. HBF is at least a year away from shipping, but flash memory manufacturer Sandisk has published fact sheets for its anticipated first-generation product. The company expects HBF to stack up to 16 NAND flash chips for a total capacity of up to 512GB per stack. It also projects memory read bandwidth up to 1.6TB/s. Sandisk's HBF roadmap also projects a second and third generation with expected read bandwidth of 2TB/s and 3.2TB/s, respectively.

What is the purpose of HBF?

Though HBF has the potential to deliver a lot more bandwidth than earlier versions of flash, you might've noticed a wrinkle. It's still a lot slower than the HBM used in high-performance GPUs. Why, then, is HBF promising?

The answer lies in key differences between AI training (teaching an LLM to predict tokens) and AI inference (serving the finished model). A model is trained by presenting it with input tokens, seeing what the model predicts, checking if that prediction was correct, and then changing weights based on the error with a step called back-propagation. While this process is simple in summary, it involves calculations across billions or trillions of model weights. That means training is heavy on both reading and writing data, which makes flash a poor fit.

However, AI inference is different. The model weights are frozen and effectively read-only, which means flash's poor write bandwidth is no longer an obstacle.

"In an inference environment massive, read-heavy data, such as the static multi-billion parameter model weights or the precomputed KV cache, can be securely housed in the HBF tier," says Kim. That would free up HBM to work as a "high-speed scratchpad."

Handy says it's a sensible way to target flash memory for inference workloads. "If you set that up right, you can get an awful lot of good performance out of that-that's just basic caching. It's one technology that I'm expecting to go places."

What's next for HBF?

Though it has potential, HBF is still early in development, and likely several years away from broad deployment. On February 25, 2026, Sandisk and SK Hynix held a kickoff event launching a joint effort to standardize HBF under a dedicated workstream within the Open Compute Project (OCP)-the same kind of open-industry body that governs many data center hardware specs. While work on the standard is ongoing, a timeline for publishing the standard has not been set.

It might seem odd for memory manufacturers-and for SK Hynix, specifically-to put forth HBF as a less expensive alternative to HBM. After all, HBM is a higher-margin product that is currently leading SK Hynix to record revenues. However, Kim frames HBF as a complementary tool rather than a rival technology.

"By alleviating the severe capacity bottlenecks of HBM without sacrificing data delivery speeds, HBF has the potential to reduce the number of individual accelerators required to run large-scale models," he says. Kim expects this will improve energy efficiency and lower costs, making it possible for data centers to further scale their AI inference hardware.

Matthew S. Smith is a freelance consumer technology journalist with 17 years of experience and the former Lead Reviews Editor at Digital Trends. An IEEE Spectrum Contributing Editor, he covers consumer tech with a focus on display innovations, artificial intelligence, and augmented reality. A vintage computing enthusiast, Matthew covers retro computers and computer games on his YouTube channel, Computer Gaming Yesterday.