Colibrì - 744B on a laptop, critically examined (and can it run gemma3:16b? No)

Colibrì (JustVugg/colibri): the 744B-model-on-a-laptop hype, critically examined - and no, it can't run "gemma3:16b"

The pitch is irresistible: "Run frontier MoE models on hardware you already own - pure C, zero deps, experts streamed from disk. Tiny engine, immense model." 25k stars, 2.7k forks, Apache-2.0, one C file per model. I dug into the repo, here's the honest picture.

What it actually is

Colibrì is an inference engine that treats VRAM, RAM and NVMe as one memory hierarchy. A 744B MoE only activates ~40B params per token, and only ~11GB of that changes token-to-token (the routed experts). So the dense part (~17B params) sits resident in RAM at int4 (~9.9GB), while the 19,456 routed experts (~19MB each, ~370GB total) live on disk and get streamed in on demand - with a per-layer LRU, a learned pinned hot-store that remembers your routing patterns, and one-layer-ahead prefetch (routing is measurably ~71.6% predictable). Think "a JIT, but for weights."

Supported families today: GLM-5.2 (744B), Inkling (975B), Kimi K3 (2.8T), DeepSeek V4 Flash (284B), OLMoE (7B). Each has its own C engine. No GPU required. Apache 2.0.

The measured reality - this is where hype meets physics

• 6× RTX 5090, full residency: 5.8-6.8 tok/s, TTFT ~13s
• 128GB CPU-only desktop: ~1.8 tok/s warm
• single RTX 5070 Ti laptop: 1.07 tok/s
• 25GB dev box (the project's origin story): 0.05-0.1 tok/s cold

That last one is the honest baseline the README itself admits. "Run a 744B model on hardware you already own" is technically true and practically a slideshow. Decode is disk-bound; your SSD bandwidth is the model speed. And the downloads are enormous: 372GB (GLM-5.2), 469GB (Inkling), ~1.6TB (Kimi K3). You need a big, fast NVMe, not just RAM.

To its credit, the project is unusually honest: "no SLA on speed, and a hard guarantee on semantics - insufficient fast memory may reduce speed; it must not quietly redefine the model." Token-exact forward validation against a transformers oracle, compressed MLA KV state (57× smaller), spec decoding that's measured and can be disabled, and they explicitly publish negative results. This is a research platform with real engineering discipline, not vaporware. The 25GB laptop origin story is legit.

The critical caveats

1. Model support is narrow and manual. Five families, each requiring a hand-written C engine. Gemma, Llama, Qwen, Mistral? Not supported. Kimi K2 / Qwen3 MoE / MiniMax are "on the roadmap" - roadmap, not done. If your model isn't in-tree, the engine doesn't exist for it.
2. Quality depends on the container. The old per-row int4 mirrors measured ~9pp worse; the gs64 container fixed it, and the MTP head must be int8 or drafts collapse to 0-4% acceptance. Quantization cost is real and measured.
3. It's a research project, not a product. No SLA on speed, greedy decode, tools/grammar only partially wired per engine. The value is the ideas (multitiering, weight-JIT, expert-atlas tooling), which are genuinely cool and are already inspiring the llama.cpp/kTransformers world.

Now the "gemma3:16b" question - no, and here's the full story

I researched it: there is no gemma3:16b. The official Gemma 3 family is 270m, 1b, 4b, 12b and 27b (Ollama tags confirm; the 27B is the big one, ~17GB at Q4). No 16B variant exists.

And even if one did, colibrì couldn't run it - for three reasons:

• Architecture mismatch. Colibrì's entire trick is streaming routed experts from disk. Gemma 3 is a dense model (with interleaved local/global attention, SigLIP vision encoder, Gemini tokenizer) - it has no routed experts to stream. A dense model has to be resident, which defeats the whole "tiny engine, immense model" concept.
• No engine in-tree. Colibrì supports exactly five families today, and Gemma is not one of them. A 16B Gemma would need a new hand-written C engine.
• You don't need it anyway. A dense 16B-class model at int4 is ~8-9GB - it just fits in RAM/VRAM directly. That's precisely what Ollama/llama.cpp already do trivially: ollama run gemma3:12b (8.1GB) or gemma3:27b (17GB) on any mid-range card. Colibrì's disk-streaming only earns its complexity at the 300GB+ frontier scale.

Fun fact: if you do want a Gemma-family MoE, that's Gemma 4's 26B-A4B - a real MoE that activates ~4B per token - but colibrì doesn't support that either.

Verdict: Colibrì is an impressive, honest research project doing genuinely novel systems work - but "frontier model on hardware you already own" currently means "374GB of downloads, a fast NVMe, and 0.1-2 tokens per second" unless you own six RTX 5090s. And for Gemma 3: run it with Ollama, not colibrì - and there's no 16B to run anyway.

Repo: https://github.com/JustVugg/colibri