I taught myself integer-only neural net training in Rust - and ran into a real quantization problem

The Problem: Dead Zone in Fixed-Point Training

A standard weight update in my integer neuron looks like this:

fn update ( & self , error : i32 , input : i32 ) -> i32 { ((( error as i64 ) * ( input as i64 )) >> 14 ) as i32 }

That >> 14 is doing the job floating-point division normally does - it scales the update down to a sane range. The problem: once the error gets small enough, (error * input) >> 14 rounds down to 0, even though the error isn't actually zero. The weights just... stop updating. Training silently stalls.

I benchmarked this "Standard" approach over 1000 runs: it converges fast, but lands with a final weight error of 77 compared to the true target weights. It gets stuck in the dead zone before it ever gets close.

My Fix: ABSL (Adaptive Bitshift Learning)

My fix was to make the shift amount adaptive to the size of the error - so updates never round away to nothing:

fn update ( & self , error : i32 , input : i32 ) -> i32 { let abs_error = error .abs (); let shift = match abs_error { e if e > 15000 => 17 , e if e > 8000 => 16 , e if e > 4000 => 15 , e if e > 1500 => 14 , e if e > 500 => 12 , e if e > 200 => 10 , e if e > 80 => 8 , e if e > 20 => 7 , _ => 6 , }; ((( error as i64 ) * ( input as i64 )) >> shift ) as i32 }

Large errors get damped more (bigger shift, prevents overshoot), small errors get amplified more (smaller shift, prevents stalling).

Stochastic Rounding Alternative

I also tried stochastic rounding as an alternative fix, which is closer to what's actually used in quantized-training resaerch. You probabilistically round the truncated fraction up or down instead of always down, so updates accumulate correctly on average.

Results

ABSL ends up the most accurate of the three, and noticeably faster per step than stochastic rounding. The trade-off: it takes more steps to fully settle, likely because it keeps making meaningful updates near the target instead of stalling out early - accuracy over speed-to-convergence.

What I'm Still Figuring Out

The shift thresholds (15000, 8000, 4000...) are hand-tuned magic numbers for this one neuron's weight scale. I don't yet know how to make them scale automatically to different layer sizes. Everything so far is tested on a single neuron, not a real multi-layer network with backprop.

Is "adaptive step size based on error magnitude" a known technique outside of fixed-point contexts? It feels conceptually close to things like Rprop, but I got here from a completely different angle (avoiding integer truncation).

Repo's here if you want to poke at the code: https://github.com/Mojo0869/green-ai

Feedback very welcome, especially from anyone who's worked with quantized or fixed-point training before - I'd love to know what I'm missing.