Compute-Optimal Is Not Cluster-Optimal

Our new paper folds the systems stage into the scaling-law stage. Price every candidate architecture on what the cluster actually delivers, and the answer changes: the sparsity an MoE 'should' have depends on the cluster you train it on.

A scaling law converts a compute budget into a model: fix the FLOPs, minimize the predicted-loss curve, read off a parameter count and a token budget. It is the most useful instrument pretraining has, and it is denominated in a currency no cluster accepts. Clusters do not bill FLOPs. They bill GPU-hours.

I have been pulling at this thread on this blog for a while. The post on model size and training horizon ended by converting compute into machine time, and the pretraining research map filed MFU-aware comparison under where scaling laws get hard: two designs can tie on loss-per-FLOP and differ on loss-per-GPU-hour, and the second number is the one on the invoice.

Our new paper with Soumajyoti Sarkar and Yuxin Tang takes that observation to its conclusion. The framework, MOSAIC, folds the systems stage into