My knowledge base missed a contradiction. My content graph found it in one query

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content What I Built Detection Debt answers a question a security operations team cannot look up: if this telemetry source goes away, which detections die and which MITRE ATT&CK techniques stop being watched? No document holds that answer. It exists only by walking connector → logTable → detectionRule → technique and subtracting sets. A keyword search over detection rules returns the rules that exist; it can never return the gap. But the thing I actually learned was something else, and it happened early. I fed two CIS benchmark PDFs and a set of Microsoft Learn pages into a Sanity Context Knowledge Base. Separately, I modelled the same claims as documents in a Sanity dataset, with references between them. Then I asked both the same question: how long should a break-glass account password be? The knowledge base gave a careful, well-sourced answer about emergency access accounts. It told me something I had missed after two manual readings of the source. It did not mention that its own corpus contains two incompatible answers. The graph returned both, in one query: | Authority | Location | Claim | |---|---|---| | CIS Microsoft 365 Foundations v7.0.0 | § 1.1.2, pp. 24-26 | at least 16 characters | | Microsoft Cloud Security Benchmark | Privileged Access, PA-5 | at least 32 characters | Same documents. Same question. One mechanism surfaced the disagreement; the other smoothed over it. The rest of this post is why - and it is not a story about either of them being worse. Demo Three questions, against the live dataset: 8 connectors, 18 log tables, 40 detection rules imported from the public Microsoft Sentinel repository, 186 ATT&CK techniques from the official STIX bundle, 8 baseline controls cited to the page. 1. What goes dark If we do not renew Defender for Endpoint, which ATT&CK techniques stop being watched? 4 tables stop. 120 GB/day of ingestion ends. 4 rules stop firing. Two techniques lose their last remaining rule: T1003 | OS Credential Dumping | was held up by DET-0020 | T1566 | Phishing | was held up by DET-0034 | T1078 and T1136 survive on other rules - and T1136 now rests on DET-0006 alone, which nobody asked about and which is the next thing to break. 2. What nothing is watching Which Credential Access techniques have nothing validated covering them? 63 of 67. Two more from the same shape of query: - 16 of 31 coverage rules read exactly one table. Over half the detection estate is a single point of failure. - 23 GB/day of telemetry that no rule reads. Defender for Identity and Defender for Cloud Apps feed four tables on the Analytics plan, and no rule - validated, tuned, draft or retired - queries any of them. I did not design that into the dataset; the agent found it. 3. Where the sources disagree How long should the break-glass account password be? Both claims, with citations, and the disagreement stated rather than resolved. This is the question that uses both Context endpoints in one turn. Ask it something else The five questions on the page are starting points, not a menu. Eleven tools sit behind it, one of which writes GROQ against the dataset, so it answers things I did not anticipate. Some that work: - How fragile is our detection coverage? - the 16 single-table rules - What's uncovered in Lateral Movement? - any of the 14 ATT&CK tactics - Can we move SigninLogs to Basic? - any of the 18 log tables - Which rule should we deploy for T1078? - ranked, and it flags that the false-positive rate it ranked by is synthetic - Should we block legacy authentication, and how? - the other contested setting: CIS names one Conditional Access policy, Microsoft documents four mechanisms - Which rules have no owner? / What's our most expensive table? / How many rules use join ? - arbitrary queries through the escape hatch And where it stops: ask about something that is not modelled - response times, incidents, analyst names - and it says it does not have that rather than filling the gap from general knowledge about Microsoft security products. That refusal is deliberate. The entire value of the tool is that it reports this estate instead of a plausible one. Every call is visible The interface shows each tool call and which of the two endpoints served it. An answer about what is missing from a security estate is worth exactly as much as the reader's ability to check it, so the checking is part of the interface rather than a debug flag. The demo runs on a free model quota of fifteen requests a minute, and one question costs a model call per agent step - so it is throttled to two questions a minute per caller. If it asks you to wait, that is the quota rather than a bug. It also runs locally with your own key from Anthropic, OpenAI or Google, and needs one environment variable to do it. Code github.com/CyberTTopic/detection-debt 122 unit assertions that need no credentials, 25 GROQ assertions against a local fixture, and 27 live checks against the real dataset that run without a model at all (npm run smoke ). The README documents three problems that each cost a day. How I Used Sanity The schema is the argument Six document types. One design rule: if a fact is an entity, it is a reference. connector ──▶ logTable ──▶ detectionRule ──▶ technique │ │ │ └── parentTechnique (self-ref) │ baselineControl ────────────────┘ └── conflictsWith (self-ref) Three of those fields do the real work: detectionRule.dataSources[] → logTable is why connector loss is computable. A rule reading one table is a single point of failure, and count(dataSources) == 1 is one line of GROQ instead of a judgement call. technique.parentTechnique , a self-reference, is what keeps coverage honest. A rule covering T1078.004 does not cover T1078. Flattening that hierarchy overstates coverage, and the only way to not flatten it is to model it. baselineControl.conflictsWith , also a self-reference, is the one this post is about: defineField({ name: 'conflictsWith', type: 'array', of: [{type: 'reference', to: [{type: 'baselineControl'}]}], description: 'Controls that govern the same setting with a different recommended value. ' + 'When a question touches a contested setting, return every claim with its ' + 'sourceAuthority and sourceLocation. Never pick one silently.', }) Every other field stores what a source says. That one stores that two sources differ, and it is the only reason the hardest question in this dataset has an answer. Field descriptions are written for the agent, not for a content editor - Sanity Context's schema_explorer surfaces them, so they are the agent's documentation. kqlFeatures explains which query constructs a cheaper Azure table plan forbids; supportsBasicPlan says a downgrade may not be offered at all. Two Context endpoints, because a Context MCP serves one mode An endpoint is either GROQ mode or Knowledge Base mode, and the mode decides which tools it exposes. I needed both, so there are two endpoints and a router. Endpoint A - GROQ mode, over the live dataset. Tools used: initial_context , schema_explorer , groq_query . This answers structure: what depends on what, how many, and above all what is absent. Endpoint B - Knowledge Base mode. Tools used: initial_context , knowledge_base_read . I pointed the Knowledge Base at three source types: - Files - the CIS Microsoft 365 Foundations v7.0.0 and CIS Microsoft Azure Foundations v6.0.0 benchmark PDFs. - Websites - specific Microsoft Learn pages on emergency access accounts, Conditional Access, session lifetime and legacy authentication. - Dataset - the tuningDecision documents from the project itself, so internal decisions are indexed beside the guidance they depart from. It produced 26 entries. The purpose field steers the outline, and getting it wrong is expensive: my first attempt used documentation directory URLs as website sources, and one of them indexed 157 pages and blew through the plan limit. Leaf pages only. The router lives in the system prompt and the agent has to name the endpoint it used: | The question is about | Endpoint | |---|---| | what breaks, what is uncovered, how many, what depends on what | the graph | | what a source recommends, why a decision was taken | the docs | | a hardening value that might be disputed | both | Why the knowledge base missed the contradiction Not because it is bad at finding contradictions. It found two others in the same build and raised them as issues I had to resolve before the index would finish. Watching three cases together is what taught me something, because the knowledge base behaved differently in each and the difference was not about how important the conflict was: Raised. An entry claimed there were ten CIS Azure activity log alert controls. Its own table, and both cited sources, listed eleven. The entry contradicted the documents it was built from, in one place, and the build caught it. Preserved but not flagged. CIS § 1.1.2 contradicts itself. Its remediation steps have you exclude a break-glass account from Conditional Access and rely on a 16-character password. A Warning at the foot of the same page states that MFA has been required for all users including break-glass accounts since 15 October 2024, and recommends passkeys instead. Both claims landed in the same entry, verbatim, side by side, and nothing marked them incompatible. Not raised at all. The 16-versus-32 disagreement. The MCSB source is in the corpus; PA-8.1 is cited elsewhere. The privileged_access entry explicitly routes PA-5 to the emergency_access entry. But emergency_access cites no MCSB source. The fact fell between two entries - and a conflict spanning two entries is not a conflict either of them can see. That is the whole lesson. A prose index can only notice a disagreement that lands inside one of its chunks. Which chunk a fact lands in is decided by an outlining pass, and nobody - including the person who wrote the purpose field that steered it - can predict tha