Show HN: Reame – a CPU inference server that gets faster as it runs

Use Cases

What Reame Is Not

What Reame is NOT for - said plainly, because trust is built here: a general-purpose ChatGPT replacement (frontier reasoning and broad knowledge need frontier parameter counts), agentic coding assistants, or creative long-form writing at scale. If your task needs a 100B-class brain, buy one; if it needs your documents processed privately, forever, at zero marginal cost - that's a realm you can own.

Key Features

• 🗂️ Persistent shared-prefix KV cache - prompt prefixes are snapshotted to disk (zstd, checksummed, LRU-budgeted) and reused across different prompts, restarts and processes. A system prompt is paid for once, by the first user.
• 📜 Palimpsest - the server remembers what it generated - every completed generation feeds an on-disk n-gram archive; future requests draft from it at zero cost. Domain workloads repeat themselves - let them pay off.
• 🎭 Il Suggeritore - grammar as a draft source - constrained decoding uses structure to forbid tokens; Reame inverts it and uses structure to propose them. List numbering, bullets and format tokens are speculated for free on content nobody has ever generated before.
• 🔮 Self-regulating speculative decoding - a small draft model or zero-cost n-gram lookup proposes tokens; the target verifies them in one batched pass. Reame measures whether speculation pays on your hardware and switches it off by itself when it doesn't.
• 🏛️ The Conclave - consensus as a quality knob - --best-of N generates N candidate answers to the same prompt in one interleaved batch (one prefill, cloned into the others via KV copy; every weight read shared) and elects the winner by majority on the final result. The moment an absolute majority agrees, the stragglers are stopped. Honestly measured: it squeezes roughly one extra correct answer per quiz out of the model you already run - it does not make a 1.5B out-reason a 3B (consensus fixes variance, not bias).
• 👥 Interleaved multi-user serving - N concurrent generations advance together inside single multi-sequence batches, sharing every read of the model weights (the cost that dominates memory-bound CPU decoding).
• 🌐 OpenAI-compatible REST API - /v1/completions, /v1/chat/completions, SSE streaming, sessions, bearer auth, metrics. Point any OpenAI client at it.
• ⚡ Zero-config CLI - reame run qwen2.5-1.5b downloads the model once, autoconfigures threads/KV/cache for the host and drops into a chat (or --serve). No config file until you want one.
• 🧪 210 isolated test cases - every layer is mockable and tested without a model; correctness of the multi-sequence, speculative and KV-clone paths is pinned against real models in integration tests. Every number below was produced by the shipped binary on the hardware named - including the negative results that shaped the design.

Benchmarks

Negative Results

Two negative results that matter. On heavily oversubscribed shared vCPUs a draft model runs as slowly as its target, so speculation is counter-productive there - Reame detects this and disables it at runtime. And the Conclave does not close the gap to a model twice the size on hard reasoning: majority voting corrects random slips, not systematic misunderstanding - we measured a 1.5B ×5 land between the 1.5B and a 3B, never above the 3B. Benchmarks that only show wins are advertising; these are engineering.

Architecture Details

Shared-prefix disk cache

Prompts are split into fixed token blocks; a chain hash keys a KV snapshot at every block boundary. A different prompt that shares a prefix restores the longest cached boundary and decodes only its own tail. Unlike GPU-resident prefix caches, snapshots live on NVMe: they survive restarts.

Self-regulating speculation

Classic Leviathan/Chen acceptance (the rejected token is resampled from the residual distribution, so the output distribution is exactly the target's), with two CPU-first twists: the draft source can be free n-gram lookup mined from the prompt itself - ideal for extraction and rewrite workloads - and a feedback controller adapts the draft length and turns speculation off when measured acceptance or draft economics go negative.

The Conclave

--best-of N submits N attempts at the same prompt to the interleaved scheduler: attempt 0 is the untouched anchor (greedy stays greedy), the explorers shift seed and heat up. The scheduler notices the identical prompts and clones the prompt KV instead of prefilling N times (copy the donor's cache, decode only the last prompt token - argmax-verified equal to a full prefill). Election is an exact-majority vote on each candidate's final number, with a Jaccard text-medoid fallback for prose; the moment a majority exists the remaining candidates are stopped mid-generation, and the CLI reports CONCLAVE consensus=k/N so a caller can escalate only when the conclave split. Use it as a quality knob: more accuracy from the model your hardware can afford, paid in idle interleaved compute rather than a bigger model's RAM.

CLI Usage

reame list                                    # model catalog + what's on disk
reame run qwen2.5-1.5b                        # download once, auto-config, chat
reame run qwen2.5-1.5b "Explain mmap"         # one-shot answer
reame run qwen2.5-1.5b --serve                # OpenAI-compatible API on :8080
reame run qwen2.5-1.5b "12*13-50?" --best-of 5 # the Conclave

run resolves a catalog name (or any local GGUF path), downloads to ~/.reame/models on first use and picks threads, KV quantization and cache directory for the host. A config file is only needed when you want control.

Installation

Homebrew (macOS / Linux)

brew tap swellweb/reame
brew install reame

Prebuilt binaries

Linux x64/arm64 and macOS arm64 on the releases page (runtime dependency: libzstd).

npm (npx reame)

Planned - binaries are already built per platform.

Build from source

git clone https://github.com/swellweb/reame
cd reame
git submodule update --init --depth 1 third_party/llama.cpp
./build.sh                                    # Release build + 210 test cases
./scripts/download_models.sh                  # TinyLlama (test model, ~670 MB)
./build/src/reame --config config/reame.conf --prompt "Hello" --max-tokens 32
./build/src/reame --config config/reame.conf --serve  # OpenAI-compatible API

Dependencies

CMake ≥ 3.16, a C++17 compiler, and for the server Boost (headers), nlohmann-json and zstd:

# Debian/Ubuntu
sudo apt install build-essential cmake libboost-dev nlohmann-json3-dev libzstd-dev pkg-config

# macOS
brew install cmake boost nlohmann-json zstd pkg-config

Configuration

[model]
path = models/qwen2.5-7b-instruct-q4_k_m.gguf
context_length = 4096           # total KV budget (shared across users when parallel > 1)
threads = 4                     # fewer is often faster on shared vCPUs - measure!

[memory]
kv_cache_type = q8_0            # f16 | q8_0 | q4_0 - halve/quarter context RAM

[speculative]
enabled = true
mode = lookup                   # model (needs draft_model_path) | lookup (no 2nd model)

[cache]
directory = /opt/reame/cache
max_size_mb = 4096              # LRU byte budget on disk

[server]
port = 8080
api_key =                       # bearer auth when set
parallel = 1                    # >1 = interleaved multi-user serving

API Endpoints

Philosophy

Reame's footprint is watt-scale, not kilowatt-scale: it targets machines that already exist and are already powered on - no new silicon is racked to serve your model. We don't claim better joules-per-token than a saturated datacenter GPU - we claim you don't need one.

Reame is young and deliberately opinionated and focused: CPU-only serving, one model per process, correctness pinned by tests at every layer. Not goals: GPU offload, training, model management UX. The llama.cpp submodule is pinned to a known-good commit and bumped deliberately. Documentation in Italian: docs/README.it.md.

Comparison with Ollama

The laptop story is the same one command: reame run qwen2.5-1.5b downloads, autoconfigures and chats - nothing to learn. From there the two projects diverge: Ollama optimizes for running many models casually; Reame optimizes for serving one workload seriously on hardware that costs nothing. The difference is one sentence: Ollama runs models. Reame remembers having run them.

General-purpose servers treat every request as brand new: compute, discard, repeat. On a GPU that's fine - compute is cheap. On a cheap CPU, compute is the most expensive thing you have, and throwing it away is the cardinal sin. Everything in Reame attacks that: the disk prefix cache, the generation archive, the grammar prompter, self-regulating speculation, interleaved multi-user batches, the Conclave. None of it exists in Ollama.

The practical consequence: a Reame server gets faster the longer it runs. The hundredth request costs a fraction of the first - the system prompt was paid once, similar answers draft themselves from the archive, structure is speculated for free. No other server has that property.

License and Sponsorship

Reame is free, MIT-licensed and built on nights and free-tier hardware. If it saves you API bills or GPU rent, consider sponsoring the work - sponsorships fund the roadmap: ARCA (the shared memory daemon), warm-ahead prefill, and first-class MoE serving.

Acknowledgments

Reame stands on the shoulders of llama.cpp (all tensor kernels; MIT). The disk-first cache thesis was inspired by antirez's DwarfStar4 line of thinking; the speculative pipeline by DeepSeek's DSpark work and the Leviathan/Chen speculative sampling theorem; archive drafting is a shipped, persistent take on retrieval-based speculation (REST); form drafting inverts grammar-constrained decoding. Ideas are cited, numbers are ours.