Seeking collaborators for scaling and independent evaluation of a new recurrent language model architecture (preprint + code) [R]

Seeking collaborators for scaling and independent evaluation of a new recurrent language model architecture (preprint + code) [R]

Hi everyone,

I've been working independently on a recurrent architecture called DABSN (Dynamic Adaptive Bias State Network) for the past several months, and I finally reached the point where I feel comfortable sharing the first preprint.

The paper is mainly about the architecture itself and its behavior on reasoning, memory, and long-sequence benchmarks (MQAR, Copy, Key-Value retrieval, A5/60, etc.). The code is also public with PyTorch, C++, and Triton implementations so everything can be reproduced.

While finishing the paper, I also trained my first language model with the same cell:

• 24M parameters
• 1B pretraining tokens
• GPT-2 tokenizer

Those results ended up being much more interesting than I expected, so I'm now writing a second paper focused entirely on language modeling, long-context behavior, and scaling.

Collaboration opportunities

This is where I'd love some help. I'm looking for people who might be interested in collaborating on the next paper, whether that's:

• independent reproduction of the results
• helping design stronger baselines and evaluations
• or having access to larger GPU clusters so we can scale the architecture much further than I can on my own

Everything I'm doing is intended to be open and reproducible from day one.

I'd really appreciate any feedback on the paper, and if the project sounds interesting, I'd love to chat. Preprint and Github are in the comments.