Head-level attention fusion trims compute

Head-Level Attention Fusion

Merging full‑attention and linear‑attention at the head granularity slashes transformer FLOPs without appreciably hurting downstream quality. The trick is to keep the expensive quadratic path only where it truly matters and let the cheap linear path handle the rest.

Before HydraHead, most hybrid designs operated at the layer level, swapping an entire layer’s attention mechanism for a linear variant or arranging a fixed ratio of full to linear layers. Those schemes struggled to reconcile the distributional mismatch between the two attention families, and researchers largely assumed the layer axis was the only practical mixing point.

HydraHead Architecture

HydraHead conserves full‑attention computation for just a quarter of the heads, delegating the remaining three‑quarters to its GDN (as described in the paper) linear module.

“By default, we retain FA computation for 25% of the heads, while the remaining 75% use the GDN structure.” [1]

Despite this aggressive pruning, the model still reaches the same benchmark scores as a traditional 3:1 layer‑wise hybrid, even when the linear‑to‑full head ratio climbs to seven‑to‑one.

“Interpretability‑based head selection enables aggressive FA compression with minimal performance degradation… matches the overall performance of a 3:1 layer‑wise hybrid even at substantially higher LA‑to‑FA mixing ratios (e.g., 7:1).” [1]

Evaluation and Open Questions

The authors evaluate HydraHead primarily on long‑context reading and reasoning suites, training on 15 B tokens before distillation. This leaves open:

• Whether the head‑selection pipeline scales down to smaller pre‑training budgets
• How it behaves on tasks that rely heavily on fine‑grained token‑level interactions, such as token‑level classification or generation with strict fidelity constraints

Performance Implications

If the reported ratios hold across typical workloads, swapping a vanilla transformer block for a HydraHead block should reduce attention‑related FLOPs by up to roughly 40% (around one‑third) while preserving accuracy, potentially enabling larger context windows or fitting larger models on edge‑class GPUs.

References

HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization