DocDrift: Your Code Drifted. Your Docs Didn't Have To.

What I Built

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

DocDrift - a background watcher that keeps your docs honest, powered entirely by open-source AI.

The Problem

Here's the loop it breaks. You ship a feature. An env var gets renamed, a default port changes, an endpoint moves to /v2. Your README is now quietly fiction, and somewhere between "I'll update it in the next commit" and "the next commit turned into 40," the docs die.

Empirical research backs the feeling - the ICSME 2023 study on outdated documentation references ("Detecting Outdated Code Element References in Software Documentation") found docs routinely pointing at code elements deleted many revisions ago.

And notably, string-matching tools can only flag references whose occurrences drop to zero - a changed default like PORT 3000 → 8080 sails straight past them, because the string still exists. DocDrift compares values and shapes, not just string existence.

What It Actually Does

• Parses the real AST, not vibes. tree-sitter WASM grammars extract exported functions, classes, interfaces, routes and config keys from your TypeScript and Python files, each with line-precise evidence. If a grammar chokes on exotic syntax, a structural fallback extractor takes over and the miss lands in a queryable failure ledger. On the docs side, your README tables, setup guides and .env.example are parsed into typed targets with the same rigor - so both sides of the comparison speak the same language.
• Runs a nine-step agent crew with two validation gates. A Mastra workflow (doc-drift-sync) chains intake → extract → docs → synthesize → gate-1 → patch → gate-2 → tone → emit into one typed, inspectable graph. Gemma drafts the findings; Gate 1 then drops every claim the AST can't prove - no phantom links survive - and Gate 2 applies the proposed patch to a shadow copy of your docs, re-parses it, and refuses edits that would break a markdown table.
• Gives you a reviewable PR, not a robot merge. The output is a unified diff, a humanized PR title and body, and a full step trace with timings. You review. That is the whole job. PRs open from a docdrift/docs-sync-<timestamp> branch against your default branch - never auto-merged.

There are two ways to use it. The Studio: sign in with GitHub (your own account, your own token - DocDrift never holds an org-wide key), pick a repo, branch and doc file, and watch the whole pipeline run live. Or the autopilot: install a push webhook on your repo, and every push that drifts the docs opens a docs-sync PR without you opening anything. HMAC-SHA256 verified, scoped to the installer's session, one user can't trigger another's webhook.

The three sloth-things baked in: it waits for your push (webhook events, not cron polling), it does the boring comparison work, and it stops before the risky part - merging is your call.

Demo

Try it in about 30 seconds at https://docdrift-rfjj.onrender.com

Code Repo: https://github.com/N-Garai/DocDrift

DocDrift was built by pair-programming with a rotating cast of AI coding agents - DeepSeek, Kilo Code, GitHub Copilot and opencode - each picked for what it was good at: DeepSeek for heavy reasoning and big-picture design, Kilo Code and opencode for rulebook-driven implementation sprints, and Copilot for inline completions and roadmap development. But the code generation was never the interesting part - the discipline was.

How I Built It

The constraint I set from day one: the open pieces have to be what makes the project work - not a coat of paint over a proprietary core. Everything below is verifiable in the repo, file by file.

Mastra as the agent harness. Four specialized Mastra Agent instances share one job: extraction, validation (Gate 1), patching (Gate 2) and tone. They're wired into a real Mastra workflow via createWorkflow / createStep - nine typed steps chained with .then() and committed as one graph. Every step's input and output is a Zod schema, so an LLM can't sneak a malformed finding past the type system, and every run produces a replayable step trace.

tree-sitter as the compiler for prose. web-tree-sitter with prebuilt grammars from @repomix/tree-sitter-wasms gives line-precise symbol extraction for TypeScript and Python. Parser failures are recorded in an in-memory ledger (last 50, surfaced at GET /api/health) and shipped to Sentry with a redacting beforeSend - cookies, tokens and session IDs never leave the machine.

Gemma as the prose engine, in a provider chain. The reasoning layer resolves per request, with a 15-second cached verdict: local Ollama Gemma (default gemma3:4b at localhost:11434) → the free Gemini API tier serving the open-weight gemma-4-31b-it → and finally a deterministic comparator engine - a rule-based drift detector (changed defaults, missing symbols, rename detection via similarity scoring ≥ 0.62, orphaned doc targets, env keys, stale routes) that produces the same artifact with zero network calls. Temperature 0.2, explicit timeouts and retries - and the chain is one env var to reconfigure.

The two gates are the anti-hallucination core. The design leans on the documentation-drift literature, all annotated in docs/RESEARCH.md: Ratol & Robillard's DOCREF ("Detecting API Documentation Errors", OOPSLA 2014) and the 2025 Concordia LLM traceability study ("Evaluating the Use of LLMs for Documentation-to-Code Traceability") both document how "phantom links" pervade LLM-linked docs. So Gate 1 re-checks every Gemma finding against the extracted symbols and drops anything the AST cannot prove before a patch is built. Gate 2 does the mirror check on the docs side: apply the edit to a snapshot, re-parse it, confirm tables stay well-formed and the symbol is genuinely documented now. Broken edits get repaired once, then flagged - never silently merged.

The whole stack is Bun + TypeScript, Express on the engine, Next.js front end, a zero-dependency JSON store for sessions and webhook registrations (no database required - swap in Postgres later behind the same interface), node:test for the test suite, and a multi-stage Dockerfile whose single Node process serves the API and the static-exported frontend on one port.

On the building side, I pair-programmed the codebase with agentic coding tools - DeepSeek, Kilo Code, GitHub Copilot and opencode - under a rulebook the whole team (human included) had to follow.

Why Does Open Innovation Matter?

DocDrift reads code, docs and env files from your repository - the payload a closed tool would demand you upload somewhere. Open makes the trust question easy to answer: Your repos stay local. The default inference path is Ollama on localhost; the deterministic engine doesn't even open a socket. Anyone can verify that by reading core/llm/provider.ts - which is exactly the point. Closed-source doc tools ask you to trust the pipeline; open-source lets you audit the pipeline.

It costs nothing to run, forever. Every component - Mastra, tree-sitter, Gemma weights, the Ollama runtime - is free, and the free Gemini tier is a fallback, not a crutch. The landing page's live demo runs with zero providers configured (the deterministic engine powers it). There's no paid plan to outgrow, no credit card, no trial clock.

The brain is swappable by config, not migration. Set OLLAMA_MODEL, point at another local model, or drop the key and use the deterministic engine. Fine-tuning how your agent behaves is an env edit.

Honest intelligence. The similarity thresholds, the evidence rules, the gate behavior - it's all in plain TypeScript with 36 tests over it rather than a marketing page. That transparency is only possible when the core is open.

My Agent Session

The build was pair-programmed with a rotating cast of AI coding agents - DeepSeek Harness, Kilo Code, GitHub Copilot and opencode - different tools for different jobs, and the interesting part was the discipline, not the code generation.

The Lazy Payoff

The honest math: every time you ship anything, keeping your own backlog of docs consistent is a 20-30 minute guilt-tax - reread the README, scan your config keys against .env.example, and see if the endpoints match. Most weeks, the honest answer is that nobody does it, and the tables fall behind.

DocDrift does not ask you to remember anything. Studio path: click through the repo picker, hit "Run drift analysis," get the diff. The demo runs on the landing page in a browser with no API key of any kind, so the price of admission to "my docs are freshly audited" is one click.

Webhook path: install once, forget. Every push lands a review PR whenever your docs fall out of date with the code. You review and merge whenever you want - DocDrift will never merge for you, which is exactly the design decision you want from a tool handling your README.

When it's wrong, it fails loudly: the mandatory evidence rules mean the diff is a real diff with real files, and the failure ledger at /api/health shows you exactly what the parser missed instead of hiding it. The sloth wins because I only do the part that needs a human: deciding whether the new table row reads well.

Prize Categories

• Best Use of Gemma - Gemma's open weights are the reasoning engine at the center of the synthesize step, with a provider chain covering both real deployment modes: local via Ollama (default gemma3:4b) on the laptop path and the open-weight gemma-4-31b-it on the hosted free tier (temperature 0.2, explicit timeouts, structured Zod outputs), plus a deterministic fallback so the pipeline never stalls without a provider.
• Best Use of Mastra - the entire brain runs on the open Mastra harness: four specialized Agent instances wired into one production createWorkflow graph (doc-drift-sync) of nine typed createSteps (intake → extract → docs → synthesize → gate-1 → patch → gate-2 → tone → emit), all with Zod-typed step boundaries and per-step traces replayable in the Studio.
• Best Use of Render - DocDrift is deployed on Render's free tier (render.yaml, Docker service): health check on /api/health, auto-deploy, production env wiring for the GitHub OAuth app and webhook secret - and a custom keep-warm service that self-pings only in production (every 13.5 minutes, cancellable with KEEP_WARM=off) so the free instance doesn't nap through its own ~15-minute idle window.
• Best Use of Sentry Agent Tracing - the engine instruments its failure paths with @sentry/node (captureParserFailure, captureWorkflowError in core/sentry.ts): every tree-sitter grammar failure and workflow error is captured with a redacting beforeSend that scrubs cookies, GitHub tokens and session IDs before anything leaves, and the same failures surface live in GET /api/health's parser-failure ledger - the observability story includes what you can see without an account.

<!-- Team Submissions: Please pick one member to publish the submission and credit teammates by listing their DEV usernames directly in the body of the post. -->