DEV Community 1h ago

Stop Making AI Agents Grind Through Huge Codebases - I Built a Deterministic Wiki Build System for Them

The Problem: Why Agents Grind Through Huge Codebases

Joining an unfamiliar codebase, what you want isn't more code - it's a wiki that explains how the thing actually works: which modules exist, where the boundaries are, how a request travels from entry to egress. The instinct for the past two years has been to throw a coding agent at it: "read this repo and write me docs." On small repos that works great. On big ones you hit three walls immediately:

Context doesn't fit - a repo with tens of thousands of files won't even fit its directory tree into a single session, let alone the implementation
Interruptions wipe everything - session crashes, machine sleeps, token budget runs out, and your progress resets to zero
No coordination for parallelism - you want to run several agents over different parts, but who splits the tasks, who takes which, who reviews the output? You, manually

And if the goal isn't "read it once" but a long-lived wiki for the whole team, add a fourth problem: code changes daily, docs never catch up, and three months later nobody trusts them.

This post introduces the open-source tool built for this problem: repowiki (on PyPI, MIT licensed). It contains zero intelligence of its own - no model APIs, no network calls. It does exactly one thing: take over the deterministic parts of "generate a wiki for this repo," so that any agent (Claude Code, Codex, OpenCode, or you yourself) can work on top of it safely, in parallel.

The design trade-off in one line: The agent supplies the intelligence; repowiki supplies the reliability.

Why the Existing Approaches Don't Work

Three routes were tried before writing any code. Each has its own lock:

Cloud AI wiki services (DeepWiki and friends): the prettiest output, but your code has to leave your machine - an instant veto for companies with confidentiality requirements. You pay per use, and the output format and hosting are a black box. A wiki that lives on someone else's cloud isn't in your git: no version history, no diff, nothing to discuss in code review.
IDE / tool built-ins (Qoder Repo Wiki, ZCode Repository Wiki, etc.): nice, but locked to one tool's ecosystem. Generation is billed in credits, and the wiki lives in the tool's own directory or platform - it can't enter CI, has no version history, and doesn't survive switching tools. There are size limits too (Qoder caps at 10k files per project). Two teammates on different IDEs means two divergent wikis.
Just letting an agent read the repo: that's the three walls above. Worse, every developer pays the comprehension cost again in every session - conclusions live in a conversation, can't be reviewed, can't be updated incrementally.

After all three, the same missing piece kept coming back: a deterministic orchestration layer. Who splits the tasks, who claimed which one, is the output acceptable, what happens after a crash? None of that needs intelligence. It needs determinism. The intelligence is already solved - pick any agent CLI. Nobody was handling the reliability part, so that's what was built.

repowiki: Reliability as a Build System

repowiki is "a build system that generates a structured wiki for any repository." The pipeline is a set of deterministic CLI commands:

plan - scan the repo, split it into per-page tasks
write - write the task catalog
next - claim the next task
--claim - a worker atomically claims the next task (no fights under concurrency)
check - programmatic validation; anchors/line numbers/H1/paths auto-repaired, only semantic defects get rejected
finalize - assemble metadata (overview page, wiki-overview, llms.txt index)
site - package a single-file offline site (~5 MB self-contained HTML)

The intelligent work - reading code, writing pages - belongs entirely to the driving agent. The CLI contains no model calls and its only runtime dependency is pyyaml.

Three direct consequences of this split:

Zero API keys, zero network calls, zero agent-CLI lock-in. Anything that "can run a shell and read/write files" can participate: Claude Code, Codex, OpenCode, or a human.
Your code never leaves your machine.
Parallelism and resumability are built in, not luck. The task catalog, claims, and heartbeats all live on disk under <repo>/.repowiki/. The CLI is a short-lived process; interrupt it anytime and continue later. Multiple agents, multiple processes, even multiple people can work on the same repo at once.

Quality doesn't depend on the agent being good; it's enforced by validation. Templates force a section skeleton, and programmatic validation plus deterministic auto-repair back it up: whatever can be mechanically fixed (anchors, line numbers, H1s, path separators) is fixed automatically; only semantic defects - hallucinated line numbers, inverted ranges, empty citations - get rejected.

A rule added in 0.7.0: if the agent writes a citation like state.py#L20-L5 (start out of range or inverted), it's no longer silently clamped - it's an error and the page gets rewritten. Hallucinated line numbers must be rewritten - the tool doesn't lie for the model.

Two Design Decisions Worth Calling Out

Task specs are self-contained; pages have zero links between them. Each task embeds the full template and style guide (~4-6k tokens), so writing any page never requires reading another page's output. That's exactly why every page task can run fully in parallel - the precondition for the multi-agent claim system below.

The output is wiki-as-code. Markdown + mermaid diagrams + file:// source citations with line ranges, committed straight into your repository: reviewable, versioned, incrementally updatable, and CI-gated like code.

The Hard Part: os.mkdir Atomicity for Concurrent Task Claims

The most interesting bit: os.mkdir atomicity, so dozens of agents can grab tasks without stomping each other.

The hard part of multi-worker concurrency is task distribution: dozens of processes reaching for work at once - what prevents duplicate claims or grabbing someone's in-progress task? A database is overkill; repowiki's answer is filesystem primitives.

Metaphor time: the task catalog is a bulletin board. To take a job, you pin your badge in the claims area - and "pinning the badge" is implemented as "create a directory with a fixed name." The OS guarantees the atomicity: fifty workers racing to create it, exactly one succeeds.

What if a worker dies? The badge carries a timestamp; workers periodically touch it to renew. Past the deadline without renewal, anyone may move the whole thing aside (rename the directory - also atomic) and re-claim.

From src/repowiki/state.py (abridged; comments are from the real source):

def _try_mkdir_claim(self, task_id: str, worker: str) -> bool:
    """Create claims/<id>/ atomically, stealing it first if stale."""
    cd = self._claim_dir(task_id)
    for _attempt in range(3):
        try:
            os.mkdir(cd)
        except FileExistsError:
            if not self._claim_stale(cd):
                return False  # live claim held by someone else
            # stale: rename it away (only one racer succeeds) and retry
            zombie = cd.with_name(f".stale-{task_id}-{uuid.uuid4().hex[:6]}")
            try:
                os.rename(cd, zombie)
            except OSError:
                continue  # another racer already stole it; loop retries mkdir
        else:
            (cd / "worker").write_text(worker, encoding="utf-8")
            (cd / "ts").write_text(now_iso(), encoding="utf-8")
            return True
    return False

Two subtle details from implementing this, for anyone building concurrent tooling:

Staleness is judged by the directory's mtime, never by the ts file inside it. The ts file is written after mkdir succeeds - if you judged by the file's time, a freshly created claim would look infinitely old and get stolen while still fresh (paraphrasing the comment on _claim_age).
PIDs are meaningless for liveness. repowiki workers are short-lived CLI processes, so "is that process still alive" can't be the heartbeat signal. The only trustworthy signal is the expiry window maintained by workers periodically touching their claim. That's also why the watch command can't "fake liveness": expired claims don't count as in-progress.

Exit codes are part of the design too: 0 success, 1 validation failed, 2 state conflict (task claimed by someone else), 3 progress-wait. Driver scripts and agents just look at the exit code - no output parsing.

The Results: Dogfooding repowiki

Dogfooding is the hardest acceptance test for this project: repowiki's own wiki is generated by repowiki itself (with an agent doing the reading and writing), so the numbers below are real.

This repo: 148 git-tracked files, ~7,300 lines of Python (tests included) → 6 chapters, 20 pages
Every page carries mermaid diagrams and source citations like state.py:343-367 - every claim in the wiki links back to the exact lines that back it. When a new teammate doubts the docs, one click shows the truth
The whole site packages into a single 4.2 MB wiki.html: sidebar navigation, scroll-spy TOC, full-text search, reading progress, prev/next, dark/light theme, and source-citation popups with syntax highlighting (a hand-written zero-dependency scanner with light and dark palettes) - the rendering libraries are inlined into the file itself. Double-click it offline, or send a colleague one file
It also exports llms.txt / llms-full.txt (the llmstxt.org convention), so any agent or IDE can read the whole wiki by index - no MCP server needed, just hand it a static file

The online sample (repowiki documenting itself, rebuilt on every push to main): wiki.repowiki - the output language follows the target repo's language, so an English repo gets an English wiki.

Even better, the CI is self-hosted too: this repo's GitHub Actions workflow runs stale --fail-if-stale on every PR - if code changed and the wiki didn't follow, the PR gets comments on the affected pages and the merge is blocked. Pushes to main rebuild the site to GitHub Pages automatically. "Docs rot" is closed off procedurally.

The full suite of 220 tests runs on a macOS / Linux / Windows × Python 3.10-3.13 matrix, with native Windows support (no WSL). Day-to-day maintenance doesn't rerun everything either: update rewrites only pages affected by the git diff, and coverage reports which files the wiki never references - the corners that still have no docs.

Getting Started in Five Minutes

pip install repowiki-cli   # or pipx; the only runtime dependency is pyyaml
repowiki skill install --agent claude   # install the agent skill (claude / codex / zcode / cursor / opencode)

Then tell your agent "generate a wiki for this repo" and it runs the whole pipeline itself.

The complete reference for all 15 subcommands, the worker loop contract, and concurrency recipes are in docs/zh/USAGE.md. Air-gapped environments are covered too: download the wheel + PyYAML from a GitHub Release and pip install --no-index on the target machine.

Closing

repowiki is at 0.8.1, MIT licensed, and carrying real work in my daily setup. The repo lives at luomsis/repowiki.

If "huge repos are unreadable, docs always rot" has been hurting you too, give it a try - and I'd love to hear from you:

How should task granularity be split for very large repos?
Does wiki-as-code + a CI gate fit your team's review flow?
What other page archetypes would you want? Pages currently follow one of six skeletons - module (default), flow, layer, data, api and event - chosen per page theme at planning time.

If it helps you, give the repo a star - for an indie open-source project that's the most direct signal, and the biggest motivation to keep going.

Found a bug or have opinions on task splitting, page templates, or the CI gate? Open an issue - I read all of them. And feel free to share this with the colleague who just inherited a giant codebase.

The boundaries are explicit (non-goals): no LLM API backend, no MCP wrapper (agents read the wiki via the llms.txt export), no resident preview server (the output is a single static file). repowiki only does the deterministic orchestration layer - the intelligence stays with your agent.

Read on DEV Community ↗ ← Back to News