RAG, CAG and KAG: Choosing How Enterprise AI Uses What the Business Knows
DEV Community

RAG, CAG and KAG: Choosing How Enterprise AI Uses What the Business Knows

A CAIO's perspective on enterprise knowledge architecture.

Full whitepaper (PDF, 29 pages, 11 figures): DOI 10.5281/zenodo.24399422.

Executive summary

Across energy, manufacturing, healthcare and financial services, I keep seeing the same pattern. The model is rarely the problem. Enterprise assistants disappoint because the right business knowledge is not in front of the model when it answers, or because nobody can prove the model was allowed to see what it used.

RAG, CAG and KAG are three ways of solving the first problem; governance decides the second. For a CAIO or CTO, choosing between them is less a technology bet than a decision about where knowledge preparation happens, who owns it, and what each correct answer costs.

RAG (Retrieval-Augmented Generation) searches a document collection when a question arrives and passes the model the few passages that matter. It is the usual starting point because it copes with large collections, frequent change and per-user permissions.

CAG (Cache-Augmented Generation) loads a small, stable body of knowledge into the model once, keeps the model's processed state (the KV cache) and reuses it for later questions. It can work well for a shared handbook that is queried all day. It does not suit large, fast-changing or permissioned content.

KAG (Knowledge-Augmented Generation) first turns documents into a graph of entities and relationships linked back to the source text, then answers by planning and running a sequence of lookups and calculations. It earns its higher cost on questions that chain several facts, depend on dates, or apply rules.

Mature programs do not pick one. They build a single governed knowledge platform and route each kind of question to the cheapest path that answers it reliably. That is the direction I recommend, and the rest of this article explains how to get there without overinvesting early.

What leaders should take from the research

Most headline numbers do not survive a reading of the papers.

CAG's famous "40x faster" is measured against re-reading the whole corpus for every question, not against RAG. Its accuracy edge over RAG is marginal and disappears as the content grows.

Graphs are an investment for specific questions, not a general upgrade. Graph methods win on questions that chain facts, dates and rules, and lose to plain retrieval on simple lookups.

Bigger context windows do not retire retrieval. Models lose accuracy well before their advertised limit, especially when the answer is not phrased like the question.

No approach is reliable on its own. Even the best systems in public benchmarks miss or get wrong a meaningful share of questions. Evaluation, and the ability to say "I don't know", belong in the design from day one.

Five recommendations for the next twelve months

  • Fix retrieval before buying anything new. Contextual chunking, hybrid search and reranking cut retrieval failures by more than half in published tests, and most production systems I review have not done all three. This is the cheapest accuracy gain available.
  • Turn on caching where the terms fit, and make CAG earn its place. Prefix caching is close to free money for stable instructions. For small, stable, shared content, run CAG side by side with a well-tuned RAG pipeline before committing.
  • Fund a graph only where real questions demand it. Look at what users actually ask. If enough of it is relational, time-bound or rule-bound, start with low-cost graph methods and reserve KAG for rule- and calculation-heavy domains such as compliance, claims or engineering change.
  • Route rather than choose. A router lets each path be justified, measured and retired on its own business case, which is how you avoid a platform built around last year's favourite technique.
  • Govern all three paths the same way. Authorize before any knowledge reaches the model, isolate caches per tenant, track where every fact came from, and report cost per correct, authorized answer to the business, not cost per token.

The picture below shows where this article lands: one governed platform in which RAG, CAG and KAG are options chosen by an orchestration layer.

Figure 1. Enterprise knowledge architecture: applications on top, an orchestration and knowledge router choosing RAG, CAG or KAG, and a governed knowledge fabric over enterprise data sources.

Part I - Concepts

1. Why LLMs need external knowledge

When I brief a board, I describe a language model as a brilliant new hire who studied everything public up to a certain date and then stopped reading: fluent and fast, but unaware of your company, your latest policies, and anything that happened since.

What a model "knows" - and what it cannot

Everything a model learned in training is stored in its weights. Researchers call this parametric memory. It has three limits that matter to every enterprise:

  • Frozen in time - What it means in practice: The model knows nothing after its training cutoff. Example: It cannot tell you last quarter's revised pricing.
  • Never saw your data - What it means in practice: Contracts, maintenance logs, SOPs, customer records were never in training. Example: It cannot answer "what is our warranty term for Product X?"
  • Cannot cite - What it means in practice: Even a correct answer arrives without a source. Example: An auditor cannot verify where the answer came from.

When a model lacks the relevant facts it will often still produce a fluent, confident answer. Unsupported answers of this kind are called hallucination.

The open-book exam

The fix is to turn a closed-book exam into an open-book one: give the model the relevant pages at the moment it answers, and ask it to answer from those pages and cite them. That is the idea behind RAG, first published by Facebook AI researchers in 2020.

Knowledge lives in a searchable index outside the model, where it can be updated without retraining. In public benchmarks, giving a model the right documents lifts accuracy substantially, yet even the best systems still miss a meaningful share of questions.

External knowledge is the difference between a model that guesses and one that grounds its answers. How you deliver that knowledge, and how you govern it, decides whether the result is enterprise-grade.

2. The three approaches, explained simply

All three approaches answer the same question - how does the right knowledge reach the model? - but they do it at different times and in different forms: RAG fetches it when asked, CAG prepares it in advance, and KAG organizes it into connections first.

An everyday analogy

Imagine three ways to help an expert answer customer questions:

RAG is the research librarian. Every time a question arrives, the librarian searches a huge, constantly updated library, pulls the three or four most relevant pages, and hands them to the expert. It copes with very large libraries and stays current as long as new and changed documents are indexed promptly. But if the librarian grabs the wrong pages, the expert answers from the wrong pages.

CAG is the open binder on the desk. For a small, stable reference - say a 150-page product handbook - the expert reads it once in the morning and keeps it open. Every question that day is answered without fetching anything. Fast and simple, but only works if the binder is small, does not change during the day, and everyone asking is allowed to see all of it.

KAG is the investigator's evidence board. Before any questions arrive, an analyst pins every person, company, contract and event to a board and draws the connections between them, each string labelled and tied to a source document. Questions like "which suppliers of Plant A have contracts expiring within 90 days?" are answered by following the strings. Powerful for connected questions, but building and maintaining the board is real work.

Figure 2 ยท the three approaches as everyday analogies. Read each column top to bottom: the difference is when the knowledge work happens - at question time (RAG), once in advance (CAG), or as a map built in advance and walked per question (KAG). The same colours mark the three approaches in every diagram that follows.

The same idea in one table

  • RAG - In one phrase: Find what is relevant. When knowledge is selected: At question time. Form the knowledge takes: Retrieved text passages. Best for: Large, changing, permissioned libraries. Main business risk: Wrong or missing answers when retrieval misses. Typical owner: Platform or search team.
  • CAG - In one phrase: Reuse what is prepared. When knowledge is selected: Ahead of time, once. Form the knowledge takes: The model's saved "reading" of the whole corpus (KV cache). Best for: Small, stable, shared references with heavy reuse. Main business risk: Stale or over-shared content. Typical owner: Platform team, with content owners.
  • KAG - In one phrase: Connect facts, then reason. When knowledge is selected: Ahead of time (the graph), then planned at question time. Form the knowledge takes: Entities, relationships and rules, linked back to text. Best for: Multi-step, relational, time-bound or rule-bound questions. Main business risk: High build and upkeep cost; wrong links in the graph. Typical owner: Data and domain teams together.

3. A better mental model: selection, reuse and structure

Instead of asking "RAG, CAG or KAG?", ask three independent questions of every request - what should be selected, what can be reused, and what must be structured - because a single answer can need all three.

Three design axes

  • Selection - The question it answers: Which information is relevant and authorized for this request? Primary technique: Retrieval: keyword, semantic, SQL, API, graph lookup (RAG). Typical failure if ignored: The model answers from the wrong or forbidden material.
  • Reuse - The question it answers: Which work can be shared across requests without breaking freshness, permissions or model compatibility? Primary technique: Caching: prompt/prefix and KV caches (CAG). Typical failure if ignored: Paying again and again to re-read the same handbook.
  • Structure - The question it answers: Which entities, relationships, time limits, rules and calculations must be explicit rather than buried in prose? Primary technique: Knowledge graphs, logical forms, deterministic operations (KAG). Typical failure if ignored: Plausible but wrong joins, dates and arithmetic.

Seen this way, hybrids are natural rather than exotic. A cached handbook does not stop the system retrieving a newly issued notice. A graph traversal does not stop it retrieving the supporting clause. A retrieval pipeline can still cache its stable instructions.

Four kinds of enterprise information

The three axes become concrete once you sort enterprise information by what it is. Most knowledge fits one of four categories, and each has a natural home:

  • Unstructured evidence - Examples: Contracts, policies, manuals, tickets, reports. Preferred treatment: Retrieval (RAG), including multimodal retrieval for drawings and scanned pages.
  • Stable shared context - Examples: System instructions, approved reference sets, a product handbook. Preferred treatment: Prefix caching, or bounded-corpus CAG where it tests well.
  • Relational knowledge - Examples: Suppliers and the parts they supply, asset hierarchies, dependencies, ownership. Preferred treatment: Graph queries (GraphRAG / KAG) or relational joins.
  • Authoritative operational state - Examples: Inventory, balances, open orders, production events, credit limits. Preferred treatment: Governed APIs, SQL and semantic-layer queries against the system of record.

The fourth category is the one most often mishandled. A vector index or a knowledge graph should never become an accidental copy of transactional data: the copy goes stale within minutes, and it bypasses the authorization rules of the system that owns the data.

Part II - How each approach works

4. Inside RAG: dynamic evidence selection

RAG is the enterprise default for grounding models in large, changing, permissioned knowledge. In my experience its quality is set far more by retrieval engineering (how documents are parsed, chunked, searched and reranked) than by the choice of vector database, which is where too many programs spend their first procurement cycle.

The pipeline at a glance

Every RAG system has two halves. An offline half prepares the library: it parses documents, splits them into chunks, converts each chunk into numbers that capture its meaning (embeddings) and stores them in search indexes. An online half answers questions: it searches those indexes, picks the best evidence, and asks the model to answer from it.

Figure 3 ยท RAG reference pipeline: offline ingestion and online serving. The top lane runs whenever documents change; the bottom lane runs for every question and snakes from the question to a cited answer. The dashed line is the only link between them: the search step looks up the indexes the offline lane built.

The four levers that move quality

The best public evidence that chunk context, hybrid search and reranking compound comes from Anthropic's 2024 contextual retrieval study. Adding chunk context, then keyword search, then reranking each reduced how often the right passage went missing, and together they cut those failures by roughly two thirds. The one-off cost of adding context was trivial with prompt caching. These were the vendor's own datasets, so test on yours, but the direction matches what I see in client pipelines: most RAG quality problems are retrieval problems, and they are cheap to fix.

5. Inside CAG: reusing a stable knowledge prefix

I treat CAG as a deployment pattern rather than an architecture: if a small, stable body of knowledge is going into the prompt anyway, compute the model's reading of it once and reuse it. Every major model vendor now offers this as a managed feature. Its practical ceiling is set by how much context a model can actually use well, not by the window size on the datasheet.

How CAG works

The pattern was named in a 2024 paper with the deliberately provocative title "Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks" (Chan et al.). It describes three phases:

  1. Preload. Fit the whole document collection into the context window, run the prefill once, and store the KV cache.
  2. Infer. For each question, load the cache, append the question, and generate. No retrieval runs.
  3. Reset. Truncate the question tokens from the cache so it returns to the clean knowledge state.

"Reset" only clears the last question. It does nothing about corpus updates: any change to the knowledge means recomputing the cache.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.