Build a Tiny Citation Gate Before Trusting RAG Answers

A RAG answer can cite a real document and still make an unsupported claim. Retrieval answers “which text was nearby?” Citation verification asks a different question: “does the cited text contain enough evidence for this sentence?” Start with a deterministic gate before adding another model.

from dataclasses import dataclass

@dataclass
class Claim:
    text: str
    citation_ids: list[str]
    required_terms: set[str]

def verify(claim: Claim, sources: dict[str, str]) -> dict:
    missing = [cid for cid in claim.citation_ids if cid not in sources]
    evidence = " ".join(sources.get(cid, "") for cid in claim.citation_ids).lower()
    absent = sorted(term for term in claim.required_terms if term.lower() not in evidence)
    return {
        "claim": claim.text,
        "valid_source_ids": not missing,
        "missing_sources": missing,
        "missing_terms": absent,
        "supported": not missing and not absent,
    }

sources = {
    "doc-1": "The service retains audit logs for 30 days.",
    "doc-2": "Enterprise plans can export logs as JSON."
}

claim = Claim(
    "All plans retain exportable audit logs for 90 days.",
    ["doc-1", "doc-2"],
    {"all plans", "90 days"},
)

print(verify(claim, sources))

Expected result: ... 'missing_terms': ['90 days', 'all plans'], 'supported': False}

This is intentionally not semantic reasoning. It catches missing references and obvious term mismatches while remaining easy to inspect. That makes it a useful first test and a poor final verifier.

Extend it carefully

• Split an answer into atomic claims. One citation at the end of a paragraph is ambiguous.
• Store source ID, URL, revision, and retrieved span together.
• Reject citations to spans the retriever did not actually return.
• Add numeric normalization so 30 and thirty can be compared.
• Only then test an entailment model, using a labeled dataset with supported, contradicted, and insufficient-evidence examples. Measure precision and recall separately.

A gate that blocks every answer has perfect recall for bad answers and no product value. A gate that approves everything is fast but meaningless.

The public MonkeyCode repository describes AI task and project-requirement workflows. Citation gates can be useful anywhere an assistant summarizes repository requirements or task evidence, but this exercise does not test MonkeyCode or describe its implementation.

Disclosure: I contribute to the MonkeyCode project. The repository description is public; this Python lesson is independent.

After this exercise, the key lesson should be clear: a citation is an address, not proof. Verification needs its own explicit step.