Learning FlashAttention the Hard Way. Part 1: The Algebraic Foundation [D]

Overview

I'm writing a short series of tutorials on FlashAttention: from theory to efficient CUDA kernels. Part 1 is the theoretical foundation. It walks through a modern algebraic formalism showing that FlashAttention is an associative operation, which lets you treat it as a regular reduction on the GPU and apply all the same scheduling optimizations.

Some recent MLSys and CVPR papers lean on this framing, and I find it much more powerful than the original.

Safe softmax, Welford's variance, and FlashAttention are the same secretly-associative operation

The twisted monoid (transport of structure), why the max-rescale coupling doesn't break associativity.

The qk_scale = log2(e)/√D you already see in FA-2 and the Triton kernels, derived from scratch

Numerical analysis

Overflow bounds, error limits, and why tiling never amplifies error.

Bird's 3rd Homomorphism Theorem as a test for whether any loop is secretly associative