Zero Forgetting in LLMs? We Benchmarked Null-Space Projection on SmolLM and Hit the Plasticity Wall

Why orthogonal projections eliminate catastrophic forgetting in autoregressive Transformers - and the mathematical reason behind the 62% downstream plasticity penalty.

The Continual Learning Dilemma in LLMs

Fine-tuning Large Language Models sequentially without retraining from scratch is notoriously broken. When you adapt an LLM to Task B after Task A, it systematically overwrites the weight configurations that enabled performance on Task A. This is Catastrophic Forgetting.

While Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA (Low-Rank Adaptation) isolate task-specific updates into small low-rank matrices:

$$\Delta W = \frac{\alpha}{r} B A$$

a critical bottleneck arises during deployment: How do you consolidate multiple sequential adapters into a single base model without incurring catastrophic interference? Directly merging delta weights:

$$W_{\text{base}} \leftarrow W_{\text{base}} + \Delta W$$

destroys historical task performance.

To evaluate this frontier, we implemented and benchmarked Wake-Sleep Null-Space Consolidation (WS-NSC) on SmolLM-135M. The code is open-sourced, and the complete benchmark data is permanently registered under DOI 10.5281/zenodo.23763908.

Here is the empirical reality of what happens when you mathematically force parameter updates into the orthogonal null-space of prior activations.

The Mechanics of WS-NSC

The framework operates in a biologically inspired two-phase regime:

[Wake Phase: Online LoRA Training] │ ▼ [Activation Manifold Collection: C = Σ (x^T x) / M] │ ▼ [Sleep Phase: Null-Space Orthogonal Projection] │ ▼ [Consolidation: W_base ← W_base + ΔW @ P_null]

1. Wake Phase (Online Learning)

   The model freezes base parameters $W_0$ and trains a standard LoRA adapter $(A_t, B_t)$ on the active task stream $\mathcal{D}_t$.

2. Manifold Estimation

   Forward hooks register linear input activations $x \in \mathbb{R}^{B \times L \times d_{\text{in}}}$ across reference tokens and historical tasks. We compute the uncentered empirical second-moment covariance matrix:

   $$C = \frac{1}{M} \sum_{i=1}^M x_i^\top x_i \in \mathbb{R}^{d_{\text{in}} \times d_{\text{in}}}$$

3. Eigendecomposition & Null-Space Projection

   We extract eigenvectors $U$ and eigenvalues $\Lambda$ such that $C = U \Lambda U^\top$. We retain the top-$k$ principal components accounting for a variance threshold $\rho \in [0.90, 0.95]$:

   $$\frac{\sum_{i=1}^k \lambda_i}{\sum_{j=1}^d \lambda_j} \ge \rho$$

   The parameter update $\Delta W$ is then projected onto the exact null-space of $U_k$:

   $$P_{\text{null}} = I - U_k U_k^\top$$

   $$\Delta W_{\text{safe}} = \Delta W \cdot P_{\text{null}}$$

   Because $x U_k \approx x$ for historical activations, projecting into $P_{\text{null}}$ ensures:

   $$\Delta W_{\text{safe}} \cdot x^\top \approx 0$$

   Theoretical guarantee: Prior task forward passes experience zero functional shift.

The Empirical Benchmark (SmolLM-135M)

We configured a 4-stage sequential continual learning setup:

• Invariant Anchor: WikiText-2 (Unsupervised language modeling)
• Task 1: GSM8K (Symbolic/mathematical reasoning)
• Task 2: IMDb (Sentiment classification)
• Task 3: AG News (Topic classification)

All experiments were executed on an A100 GPU using PyTorch AMP across all 7 projection matrices (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).

Benchmark Metrics

Three Core Technical Takeaways

1. Catastrophic Forgetting is Completely Neutralized

   In direct merging, catastrophic forgetting is glaring: backward tasks degrade by $+5.83%$, and the general-domain anchor drifts by $+0.057$ nats. Under WS-NSC ($\rho=0.90$), backward forgetting drops to $-0.39%$ and anchor drift drops to $-0.005$ nats. Historical knowledge is preserved without storing or replaying raw pretraining datasets.

2. The Severe Plasticity Tax

   While stability is solved, it incurs an acute functional cost. Under $\rho=0.90$, $36.2%$ of the linear subspace is locked down, causing post-sleep task performance to retain only $37.7%$ of what was acquired during the wake phase. Pushing protection to $\rho=0.95$ locks $54.5%$ of the rank, collapsing retained gains down to $24.4%$. In deep nonlinear architectures, hard orthogonal projection chokes forward model capacity.

3. Weight Norm $\neq$ Functional Capability

   A common assumption in model merging is that preserving weight magnitude preserves task utility. Our data disproves this: retaining $73.9%$ of parameter update norm resulted in retaining only $37.7%$ of task gain. Spectral truncation clips the highest-energy activation vectors needed for downstream task execution.

Reproducibility & Open Source

The full codebase, automated plotting pipelines, and raw execution logs are open for external verification:

• GitHub Repository: https://github.com/AkshatRaj00/ws-nsc
• Permanent Scientific Record (DOI): 10.5281/zenodo.23763908

To reproduce the benchmark locally:

git clone https://github.com/AkshatRaj00/ws-nsc.git
cd ws-nsc
pip install -r requirements.txt
# Run empirical sequential evaluation
python src/ws_nsc.py --model HuggingFaceTB/SmolLM-135M --amp --batch-size 4 --eval-batch-size 4 --var-retention 0.90 0.95
# Generate loss trajectories & Pareto frontier plots
python src/plot_results.py

Citation:

@article{raj2026wsnsc, title={Wake-Sleep Null-Space Consolidation: Quantifying the Stability-Plasticity Frontier in Sequential LLM Adaptation}, author={Raj, Akshat}, year={2026}, publisher={Zenodo}, doi={10.5281/zenodo.23763908}, url={https://doi.org/10.5281/zenodo.23763908} }

Dev.to / Medium Tags to Use: machinelearning, datascience, ai, deeplearning, python