A Folder of Docs Is Not a Knowledge Base

Disclosure

Disclosure up front: I build agentproto and its corpus tooling, which is what the walkthrough uses. The commands are real and checkable; the problem in the first half is one every RAG setup hits. Corrections welcome - file an issue.

The Trap

The advice is everywhere and it sounds complete: give your agent your knowledge - just do RAG over your docs folder. So you point a retriever at docs/, embed the chunks, and wire it up. The agent starts answering questions about your systems.

Then you ask it something that matters, and it gives you a fluent, confident paragraph - and you have no idea whether it's true, because you have no idea where it came from. Was that from the current runbook or a design doc you abandoned in 2024? Is that retry number the real one, or a chunk from a stale README that happened to score well on cosine similarity?

You can't tell. Neither can the agent.

That's the tell that you don't have a knowledge base. You have a search index over a pile of documents, which is a different and much weaker thing.

The One Idea

If you remember nothing else: A folder of docs is not a knowledge base. What makes it one is provenance - every fact knowing which source it came from, so the agent can cite it and you can check it.

The Two Things a Doc Dump Can't Do

A folder of markdown fails as a knowledge base for two structural reasons, and no amount of better embeddings fixes either.

It retrieves prose, not facts

A document is a wall of sentences about many things at once. Chunk it and you get arbitrary slices - half a paragraph about retries glued to the start of a paragraph about logging. The agent gets near the answer and reconstructs the rest, which is exactly where confident-but-wrong comes from. The unit of knowledge should be a claim, not a 512-token window.

It forgets where everything came from

Once a chunk is embedded, it's just a floating string. There's no thread back to "this is from runbooks/payments.md, last touched in the April incident." So the agent can't cite, and - worse - you can't audit. A knowledge base you can't audit is a knowledge base you can't trust, which means it's a liability the first time it's confidently wrong.

Both problems have the same fix, and it's not a fancier retriever. It's a pipeline: turn sources into atomic, sourced claims before anything queries them. That pipeline is what separates a corpus from a folder.

What's Your "Knowledge Base" Actually Made Of?

Place yourself before you build. Four honest answers.

Nothing

The agent runs on pre-training and whatever you paste. Generic by default. Ceiling: it never knows one thing that isn't already public.

A docs folder + RAG

You retrieve chunks by similarity. Better than nothing, and it's the trap this piece is about: no provenance, no dedup, no notion of a claim - it answers from sources it can't name. Ceiling: confident, unverifiable, and stale the day a doc rots.

A hand-curated wiki

Real, human-maintained, often good - but it's written for people to read, not for an agent to query claim-by-claim, and it goes stale between the edits nobody makes. Ceiling: high quality, low queryability, manual upkeep.

A distilled, sourced corpus

Sources imported with their origin attached, distilled to claim-level entries, each carrying its provenance, queryable by topic. This is the one an agent can cite and you can audit - and it's four commands.

Where are you? If your honest answer is "RAG over a docs folder," you have the confident-but-unverifiable setup - the most dangerous kind, because it feels like it works right up until it states a stale fact with a straight face.

The rest of this turns the pile into a corpus.

Step 1: Import, and Keep the Receipt

Start by pulling your sources into a corpus workspace - and the non-negotiable is that each source keeps a thread back to where it came from.

npm i -g @agentproto/corpus-cli

# scaffold an AIP-10 workspace (generic research preset, no seed noise)
corpus init house-knowledge knowledge/house --preset research

# import your sources; --tags is what makes the corpus queryable by topic later
corpus import-web knowledge/house --urls-file runbooks.txt --tags runbooks
corpus import-web knowledge/house --urls-file incidents.txt --tags incidents

Every imported source lands as an AIP-10 record carrying its originalUrl - the provenance thread. An article comes in via readability; a walled page routes through a browser; a recorded incident review or an architecture talk on video comes in via transcription - same workspace, same provenance, so a decision your team only ever said out loud becomes a citable source.

Import is resumable: re-run it and it skips what's already in, so you grow the corpus instead of rebuilding it.

Step 2: Distill - Turn Documents into Claims

This is the step a folder-and-RAG setup skips entirely, and it's the one that matters. Distillation reads each source and emits atomic, claim-level entries, roughly seven per source, each one carrying the source it came from.

# cheap + strong: distill on an open model via OpenRouter (saves frontier credits)
corpus distill knowledge/house --engine opencode -m openrouter/deepseek/deepseek-v4-pro

Now the unit of knowledge is a claim, not a chunk - "payment retries are capped at 3 with an idempotency key, per incident 2026-04" is one entry with one source, not a slice of a paragraph that happened to embed nearby. When an agent retrieves this, it retrieves the fact and its origin, which is what lets it answer "per incident 2026-04…" instead of "a common approach is…". Provenance stops being metadata and becomes the answer.

Why claims beat chunks, grounded

Anthropic's context-engineering guidance is to keep the working context small and retrieve high-signal facts just in time rather than pre-loading walls of text. You can't do just-in-time retrieval on a document - it's too big and says too much. You can do it on a claim. Distillation is what makes your knowledge retrievable at the right grain.

Step 3: Gate the Corpus - A Knowledge Base Is Maintained, Not Filled

A pile of docs is fill-once-and-forget. A corpus is an artifact you keep, so it gets a quality gate before anything trusts it.

corpus validate knowledge/house        # schema-valid sources, 0 errors
corpus lint knowledge/house            # structural checks
corpus knowledge knowledge/house --tags incidents --max 20  # preview what an agent would pull

Two failure modes to catch here, because they're the ones that poison a corpus quietly:

• A bad scrape - a retriever grabbing a cookie-wall or a nav menu instead of the article - still gets distilled into confident-sounding entries; grep for self-flags ("content does not match", off-topic nouns) and quarantine them into a demoted/ folder so they never reach an agent.
• Coverage - for each topic you care about, does the corpus actually have sourced claims, or is that tag empty? A gap you find now is a gap the agent won't paper over with a guess later.

This is the discipline the doc-dump never had: your knowledge base is only as good as its worst source, so you check the sources.

Step 4: Serve It - Every Agent, Claim-Level, Sourced

Now the corpus becomes a tool. Back it with a knowledge.search driver pointed at the workspace you just built, and serve it over MCP - the full contract-and-driver pattern is its own walkthrough, but the payoff is one environment variable and one command:

CORPUS_WS=knowledge/house agentproto serve
# serves knowledge.search to every agent

Every agent that mounts the daemon - Claude Code, Codex, a cheap local model - now queries the same distilled, sourced corpus, and because the driver reads local files, your knowledge never leaves the machine. Ask any of them your retry policy and the answer comes back with "incident 2026-04" attached. That citation isn't decoration - it's the thing that lets a skeptical reviewer (human or agent) check the answer against the source instead of trusting it.

The Honest Edges

A corpus is a real artifact, so it has real upkeep and real failure modes.

• A wrong corpus is worse than no corpus. A stale claim gets retrieved with the same confidence as a fresh one - now with a source attached, which makes it more persuasive and more dangerous. Provenance is what saves you: because every claim names its origin, a wrong answer is traceable to the stale source and fixable, instead of an anonymous hallucination you can't hunt down.
• You maintain a corpus like you maintain docs, because that's what it is.
• Don't let the agent write your knowledge base. A corpus distilled from real runbooks, incidents, and decisions is yours; one an agent generates about itself is the internet's average with your repo's name on it. Distill from sources you'd stand behind - the whole point is that the knowledge is earned, not synthesized.

The fair comparison

Managed RAG products and vendor retrieval are less setup - if you want "upload your docs, get a chatbot," they'll get you there faster, and for some teams that's the right trade. What you give up is the two things this whole piece is about: claim-level provenance you can audit, and a knowledge base that lives in your repo, on your machine, not in someone's index. If your knowledge is your edge, those aren't details.

Build the Corpus, Not the Pile

Everyone rents the same models now, so the one part of the stack that's genuinely yours is what your agent knows - and that's only true if what it knows is real.

A folder of docs behind a retriever gives you fluent answers from sources you can't name and can't check, which is the exact opposite of what a knowledge base is for.

The fix isn't a better embedding model. It's a pipeline: import your sources with their provenance intact, distill them into claims small enough to retrieve and honest enough to cite, gate the result like the artifact it is, and serve it to every agent you run.

Four commands, and your agent stops guessing from a pile and starts answering from a corpus - one where every fact can tell you exactly where it came from.

Give your agent knowledge. Just make sure it's the kind that can show its work.

If you build your knowledge base a way that keeps provenance without the distill step - or you've made folder-RAG genuinely auditable - tell me where. I'll fix the piece.

Written by the maintainer of agentproto (Apache-2.0, source) - a local, cross-vendor daemon for running and governing coding agents. Same contract as our /compare page: dated facts, named strengths, corrections by issue. Got something wrong? File an issue.