Seven months of self-hosting our own AI stack: four bugs I can point at in the changelog

We run our team's AI stack on our own hardware. The weekly development log starts the week of January 19, 2026, the repository was created on February 3, and today it sits at roughly 2,100 commits under MIT. I want to write about four things that broke, because every one of them is in the public changelog and you can check my work. No war stories I can't back up.

Setup

A Mac mini runs the application: API and web under PM2, with PostgreSQL, Redis, and the sandboxed agent, MCP, and artifact processes in Docker. An NVIDIA DGX Spark GB10 next to it runs vLLM, plus BGE-m3 for embeddings and FLUX for image generation. The two are connected over a private Tailscale link. That is our setup, not a requirement. Any OpenAI-compatible endpoint works.

The default local model is qwen3.8-27b served through vLLM behind a LiteLLM proxy, with a 262K context window. External providers (OpenRouter, NVIDIA NIM, Ollama) only enter the picture if you register your own key, encrypted at rest with AES-256-GCM. Register nothing and every model call stays local.

Bug 1: the application limit was not the real limit

We accept large files as inputs to agent tasks. The application will happily allow a file well past 100 MB. That turns out not to matter. Cloudflare documents a 100 MB maximum upload size for Free and Pro zones, so a single multipart request was rejected at the edge with HTTP 413 before it ever reached the API. No validation of ours ran. No useful progress was shown.

The fast workaround would have been an unproxied upload host. We kept the protected public route and changed the request shape instead: a four-step chunked protocol with a one-time claim, authentication on every operation, chunk writes that are safe to retry, and reuse of the existing storedPath contract after assembly so extraction and cleanup don't fork.

The lesson is not subtle, but it cost us anyway: your app's configured limit is a claim about your app, not about the path a request actually takes.

Bug 2: a single static line pinned us to a model that no longer existed

On September 2 the DGX swapped models. The app stayed bound to the old qwen3.6-35b-a3b name, because one static catalog line was the only source of the model list. Boot and periodic probes now read LiteLLM's /model/info to discover local models. The static list survives only as a fallback. Default is qwen3.8-27b (1.38.0).

If you run a gateway, ask yourself where your model list actually comes from. Ours came from a constant, and constants do not notice when the GPU host changes its mind.

Bug 3: fan-out met real rate limits

Running Discussion and Deep Research against external models hit per-minute limits on free and developer keys. The measured result: five parallel expert calls came back 5/5 with 429 on a B.AI free key, and 3/5 on a hasa key.

The fixes, all in 1.40.0:

• The external execution client is wrapped in a per-provider semaphore with exponential 429 backoff that honors Retry-After.
• Deep Research fan-out concurrency and timeouts follow the provider hint.
• The SDK's own blind retries were set to zero, with a multiplier on its timeout, so a long-reasoning model is no longer cut off every 360 seconds.
• An explicitly chosen external model that cannot run now surfaces an error instead of silently falling back to local.

That last one matters more than it looks. Silently substituting a different model is the kind of helpfulness that destroys trust.

Bug 4: overflow, handled rather than hidden

We run a 262K context window and still hit it. On entry, prompt tokens are estimated, images included. If the window is exceeded, input is truncated, then max_tokens is reduced, and in the extreme a ContextOverflowError returns HTTP 413 with an audit record and an automatic webhook alert.

Related, and from the same release: per-request prompt image totals are now capped at vLLM's own --limit-mm-per-prompt limit of 8. We were enforcing that in one place and not the other.

The security review I would rather not publish

On September 2 we ran a security review of apps/api. The first batch of findings, closed in 1.39.0:

• System skills could be overwritten by any authenticated user. That is stored prompt injection into every user's prompt.
• Anyone could assign their own skill to a shared industry agent.
• A push subscription could be registered under another user's id, receiving copies of that user's notifications.

Pinning the owner to req.user fixed all three. It also fixed the fact that web push had never actually been delivered at all.

The batch further covered an IDOR on internal bundle installation, enforcement of the high-risk MCP tool role gate on the execution path, CSV formula injection in exports, an existence oracle on other users' MCP server status, ownership checks that passed on empty values, and an advisory lock around first-run administrator setup.

I am publishing this because a changelog that only contains features is not a changelog.

What is actually in the box

• Agent tasks. Multi-turn runs in a persistent Docker sandbox with shell, Python, browser, and files, behind human approval. Each run reports turns, elapsed time, and token cost. When a goal is not met, it emits a [GOAL_INCOMPLETE] marker and a goal judge instead of marking itself done.
• Deep research. Fan-out search, source fetch, claim verification, cited synthesis.
• Discussion mode. MoA-style: picks specialists, gives them the same evidence, runs them in parallel, synthesizes. Off by default and switched on per message.
• Per-role model routing. agent, judge, research, spawn, review, summary each resolve their own model, per-user or admin-global, failing open to the local default.
• 22 built-in MCP tools plus external servers, each isolated in Docker with --cap-drop ALL, non-root, memory caps, a network policy, and realpath-guarded mounts.

Limits, stated plainly

• The desktop app is macOS on Apple Silicon only. The web UI runs anywhere.
• No Kubernetes story, and none planned. Single host by design.
• Guests on the hosted demo get the default local model only.
• Self-hosting needs Node 24, PostgreSQL, and Docker. Setup time is mostly however long your model endpoint takes.

It's public

Source is MIT, so you can read it before you run it. The weekly development log is on the site, including the weeks that went badly. One week is missing from the archive entirely: W18 had zero commits, and an empty week is left out rather than written up.

• Source: https://github.com/openmake/openmake_llm
• Live demo: https://chat.openmake.cc
• Self-host guide: https://openmake.cc/en/docs/
• Dev logs: https://openmake.cc/en/blog/

If you run your own stack, I would like to hear which layer lied to you first.