My README Promised Flags My CLI Doesn't Have. I Built a Drift Detector.

A stranger opened an issue on one of my small CLI tools last month: "The --watch flag in your README doesn't exist." They were right. I had removed --watch two releases earlier, rewritten the feature as a config option, updated the changelog, and forgotten the README's usage section entirely. Worse, the README still showed a fenced code block demonstrating the flag, so every new user was copy-pasting a command that errored out immediately. This is documentation drift, and it's embarrassingly common. Code changes continuously; prose changes when someone remembers. The twist is that my README also contained correct examples that had merely been reworded - so a naive string diff between "flags mentioned in the docs" and "flags in --help output" produces a pile of false positives alongside the real bugs. That combination - mostly mechanical matching, with a fuzzy residue that needs judgment - turned out to be a sweet spot for a free AI model. Not to write documentation (I don't trust generated docs), but to reconcile it: decide whether a prose claim is contradicted by actual CLI behavior. Why a free model fits this specific job I run the check on every release, so per-token pricing would be annoying, and the task is genuinely low-stakes: the deterministic part of the tool catches the clear violations, and the model only arbitrates ambiguous prose claims, which a human then reviews anyway. I'm using MonkeyCode's free model access for the fuzzy-matching step - it exposes an OpenAI-compatible chat endpoint, which meant the script below needed no SDK beyond the standard library. If your docs describe internal tooling you can't send to a third party, they also have a free server option for self-hosting; the script treats the endpoint as configuration for exactly that reason. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Two things a free model is not doing here: it is not the source of truth (the --help output is), and it is not allowed to fail silently (every answer is validated against an allowlist before being reported). The artifact: driftcheck.py The pipeline has three stages, and the order matters: - Extract claims. Pull flags out of fenced code blocks in the README. - Diff deterministically. Compare them against flags parsed from real --help output. Set difference, zero intelligence required. - Reconcile prose. Only for the remaining ambiguous claims (sentences like "supports recursive watching"), ask the model whether the help text supports, contradicts, or can't confirm them. #!/usr/bin/env python3 """Detect drift between README claims and actual CLI behavior. Usage: python3 driftcheck.py README.md -- ./mytool Env vars: DRIFT_BASE_URL OpenAI-compatible endpoint (e.g. MonkeyCode) DRIFT_MODEL model name your provider currently exposes """ import json import os import re import subprocess import sys import urllib.request BASE_URL = os.environ["DRIFT_BASE_URL"].rstrip("/") MODEL = os.environ["DRIFT_MODEL"] FENCE = re.compile(r"(?:bash|sh|console)?\n(.*?)", re.DOTALL) LONG_FLAG = re.compile(r"--[a-z][a-z0-9-]+") HELP_FLAG = re.compile(r"^\s+(?:-[a-zA-Z],\s+)?(--[a-z][a-z0-9-]+)", re.MULTILINE) def help_text(prog: str) -> str: out = subprocess.run([prog, "--help"], capture_output=True, text=True, timeout=15) return out.stdout + out.stderr def claimed_flags(readme: str) -> set: flags = set() for block in FENCE.findall(readme): flags.update(LONG_FLAG.findall(block)) return flags def real_flags(help_out: str) -> set: return set(HELP_FLAG.findall(help_out)) def prose_claims(readme: str) -> list: """Sentences that assert capability but live outside code blocks.""" body = FENCE.sub("", readme) verbs = ("supports", "can ", "allows", "automatically", "detects", "handles") return [s.strip() for line in body.splitlines() for s in re.split(r"(? str: """Ask the model; accept ONLY a one-word verdict from the allowlist.""" prompt = ( "CLI --help output follows:\n\n" + help_out[:12000] + "\n\nClaim from documentation: "" + claim + ""\n\nAnswer with exactly one word: SUPPORTED, CONTRADICTED, or UNCLEAR." ) body = json.dumps({ "model": MODEL, "messages": [{"role": "user", "content": prompt}], "temperature": 0, }).encode() req = urllib.request.Request( f"{BASE_URL}/chat/completions", data=body, headers={"Content-Type": "application/json"}, ) with urllib.request.urlopen(req, timeout=90) as resp: verdict = json.load(resp)["choices"][0]["message"]["content"].strip().upper() for word in ("SUPPORTED", "CONTRADICTED", "UNCLEAR"): if word in verdict: return word return "UNCLEAR" # never trust an unexpected answer; downgrade it def main() -> None: readme_path, prog = sys.argv[1], sys.argv[3] readme = open(readme_path).read() help_out = help_text(prog) phantom = claimed_flags(readme) - real_flags(help_out) print("## Phantom flags (in README, not in --help)") for f in sorted(phantom): print(f"- {f}") print("\n## Prose claim reconciliation") for claim in prose_claims(readme): print(f"- [{reconcile(claim, help_out)}] {claim}") sys.exit(1 if phantom else 0) if name == "main": main() Design choices worth stealing: - The model is the last resort, not the first pass. Every flag discrepancy is found by pure set arithmetic. That's deliberate: deterministic checks are free, instant, and can't hallucinate. The model handles the 20% of claims that are sentences, not flags. - The verdict space is closed. reconcile() downgrades anything that isn't one of three words toUNCLEAR . If the provider swaps the underlying model tomorrow, the worst case is moreUNCLEAR rows for a human to skim - never fabricated confidence. - Exit code is driven by the deterministic stage only. I'd never let a language model decide whether my release CI turns red. What it found in my own repos I ran this across my three public tools and hand-verified every report: | Finding type | Reported | Real bugs | False alarms | |---|---|---|---| | Phantom flags | 9 | 7 | 2 (typos in --help itself - also bugs!) | | CONTRADICTED prose | 5 | 4 | 1 | | UNCLEAR prose | 11 | - (human-reviewed, 3 were real drift) | - | Two observations stood out. First, the "false alarms" for phantom flags were cases where the help text had the typo and the README was right - drift works both directions, which I hadn't considered. Second, the model's value concentrated entirely in the prose stage: it correctly flagged "automatically detects your config format" as contradicted after I'd removed auto-detection, something no regex could have caught. Limitations, and who shouldn't bother - It only checks what --help reveals. If your tool's real behavior lives in subcommands, config files, or runtime side effects, extend the extraction stage first; the reconciliation stage can't help you. - Prose reconciliation is advisory. CONTRADICTED means "read this sentence yourself," nothing more. About a quarter of its verdicts in my run needed correction. - Free endpoints throttle. Three repos is fine; a 40-package monorepo README sweep in parallel CI will hit rate limits. Run it nightly, not per-commit, or use a self-hosted option for volume. - Skip this entirely if your README is five lines, if your CLI is generated from the same source as your docs (e.g., you already generate docs from argparse definitions - do that instead, it's strictly better), or if you expect it to validate tutorials, where correctness means "the commands run," not "the flags exist." The takeaway that generalizes The useful reframe for me was treating the model as a reconciliation layer between two machine-readable-ish sources of truth, not as a writer. Docs vs. help output is one pair; the same pattern fits schema vs. example payloads, or changelog vs. actual exports. Wire an OpenAI-compatible endpoint behind two env vars - I pointed mine at MonkeyCode - keep the deterministic stage in charge of the exit code, and hand-verify one batch of reports before you trust any of them. If you build this for a different pair of sources, I'd genuinely like to hear which one in the comments. Top comments (0)