HyperSAE: Decoupled Poincaré Geometry for Sparse Autoencoders -- 9.8% MSE reduction, 0.2% dead latents on Gemma-2-2B [P]

I built HyperSAE, a PyTorch library that applies Poincaré hyperbolic geometry to Sparse Autoencoders for mechanistic interpretability.

• GitHub: https://github.com/vishal-dehurdle/hypersae
• Paper: https://vishalvermalabs.com/papers/empirical-validation-hypersae-poincare-geometry/
• Install: pip install hypersae

Motivation

Standard SAEs embed dictionary atoms in Euclidean space, where volume grows as O(r^d). The concepts LLMs learn form branching hierarchies that expand as O(b^r). At 16K+ dictionary sizes, this mismatch causes feature collisions at the boundary, dead latents, and reconstruction degradation.

Architecture

HyperSAE uses a decoupled dual-speed design:

• Forward pass remains entirely Euclidean. Zero inference overhead.
• Causal steering stays a single vector addition.
• During training, dictionary weights are projected into the Poincaré ball.
• An entailment cone loss organizes parent concepts near the origin and child concepts near the boundary, where hyperbolic volume expands exponentially.

Results

Gemma-2-2B Layer 13, 20M tokens FineWeb-Edu, NVIDIA L4.

Library

The library includes co-activation queue tracking, TriPartite loss (reconstruction + L1 sparsity + entailment), and a single-class trainer interface. Feedback on the geometric formulation or benchmark methodology is welcome.