Testing whether discrete topology is a more durable memory medium than continuous weights - it isn't ... probably [R]

I ran a controlled comparison of two "write-in-place" memory adapters for a frozen LM: one edits continuous weights, the other (EPMem) keeps weights frozen and random and only edits a binary edge mask via straight-through estimation. Storage bits and write steps are matched across both.

Benchmark Results

On a continual fact-writing benchmark (TinyStories-33M), EPMem writes facts fine but forgets them just as fast as the continuous adapter, underperforms it overall, and takes more write steps to get there.

Mechanism

Mechanistically: per-write interference scales as sqrt(F) in the number of mask flips F, and retention decays faster than a random-collision null - so the mask-learning gradient isn't colliding by chance, it's preferentially re-flipping edges that are already storing something (anti-consolidation, not neutral overwrite).

Replication at Scale

Replicated on Pythia-410M: same pattern, sharper - scale makes writing easier, retention worse.

Takeaway

Binary-mask methods that do avoid forgetting (SupSup, WSN, etc.) work because they explicitly enforce non-overlapping allocation between writes, not because discreteness itself is forgetting-resistant. The substrate isn't the lever - the write/allocation rule is.

Paper and Discussion

Paper: https://zenodo.org/records/22071715 (DOI: 10.5281/zenodo.22071715)

Genuinely interested in pushback - particularly whether this holds outside fact-writing tasks, or whether it's an artifact of straight-through estimation specifically.