The calibration bug that taught our fraud agent to wave fraud through

Building an agentic fraud investigator on TigerGraph - and the two measurement mistakes that silently inverted it. All 18 architecture diagrams, interactive: FraudGraph Blueprints: system, agent loop, MCP path, schema, GraphRAG, policy rules, all twenty benchmark cases, evaluation and failure modes. The architecture at a glance Three triggers, one LangGraph agent, TigerGraph reached through the MCP server, and a deterministic core the LLM cannot reach. The one conditional edge that makes it an agent: gather more evidence, or act. The problem, stripped down TigerGraph's Hacker House Goa challenge hands you six months of card transactions - 590,742 of them, from the IEEE-CIS dataset - and takes away the one column everyone reaches for. There is no isFraud flag. Every transaction carries a risk score from the bank's own model instead, and the dataset README is blunt about what that score is worth: Above 0.7, most flagged transactions turn out to be legitimate. Some fraud scores near zero. What you get instead of labels is four months of closed investigations - 5,565 of them, confirmed fraud and cleared false alarms, with the analyst's notes. And twenty new alerts to decide. So the task isn't classification. It's investigation: work out what kind of fraud this is, how far it goes, what to do about it, and when you know enough to act. That last clause is the whole thing. The shape that the submission format forces Buried in the answer format is a requirement that quietly determines the architecture: The next best action and required approval route recorded: before any additional evidence is requested, after any additional evidence is received. You cannot produce that pair honestly from a single-shot pipeline. There is no "before" unless the agent genuinely commits to a recommendation under uncertainty, then decides what it needs, then revises. So the topology is: retrieve → believe → is this defensible? │ no ▼ PROVISIONAL action + approval route │ pick the question worth asking │ response │ ▼ REVISED action + what changed Everything else follows from wanting that loop to be honest. The split: what may hallucinate, and what may not The model sits outside the decision boundary; people sit on the approval boundary. Every decision a bank would have to justify to a regulator lives in a core/ package that cannot import an LLM. Not "does not" - cannot. A test walks the AST of every file in that directory and fails the build if an import of anthropic , openai , langchain or anything with llm in the name ever appears: def test_core_cannot_import_an_llm(): banned = {"anthropic", "openai", "google", "langchain", "langgraph", ...} for path in (ROOT / "src" / "fraudagent" / "core").glob("*.py"): tree = ast.parse(path.read_text()) ... assert not offenders, f"core/ must stay LLM-free: {offenders}" That covers the Bayesian ledger, the policy engine, the approval routing, the stopping rule and the episode reconstruction. The model plans which graph primitives to run, synthesises evidence into prose, writes the case summary and the SAR narrative. If it proposes an action anyway, the policy decision object drops it before it can reach an answer file. The practical consequence, which I like more than the principle: the agent runs end to end with no API key at all. Verdicts, probabilities, actions and approval routes are bit-identical; only the wording changes. Anyone can clone the repo and reproduce the twenty answer files without buying anything. What TigerGraph is actually for here Similar past cases found by meaning and by shared entities inside one GSQL query, then fused. The obvious answer is "traversal", and that's true - but the reason this project belongs on a graph database with vectors in the same store is a query that needs both at once. Case memory is the requirement: prior closed cases must measurably change what the agent concludes. The naive implementations both fail in the same characteristic way: - Vector search alone returns prior cases that sound similar. Narratives are templated, so it happily returns five cases about card testing when you asked about a different burst pattern. - Traversal alone returns cases that are connected - same card, same device, same customer. Precise, and usually empty. Neither is what an analyst means by "have we seen this before". So fuse them, in one GSQL statement, inside the database: candidates = vectorSearch({ClosedCase.narrative_emb}, query_vec, k * 8); candidates = SELECT c FROM candidates:c WHERE c.opened_at c.@distance); linked_d = SELECT c FROM seed_dev:v -(CC_DEVICE_OF:e)- ClosedCase:c WHERE c.opened_at 1); result = SELECT c FROM (candidates UNION linked_t UNION linked_c UNION linked_d):c ACCUM c.@score += 0.55 * @@vec_score.get(c) + 0.45 * (1.0 - exp(-0.6 * @@shared.get(c))) ORDER BY c.@score DESC LIMIT k; Two details do real work. The saturating term on the graph leg means the first shared entity is worth a lot and the fourth almost nothing - with a linear term, one busy device profile dominates every ranking. And the as_of guard on both legs: a memory system that can retrieve a case opened after the one it's reasoning about isn't a memory system, it's a leak, and every number downstream of it is fiction. There's a second embedding space too, and it's the one I'd defend hardest. V1 -V339 are Vesta's engineered features: real signal, no names, and utterly hopeless as LLM context. 339 unnamed columns isn't evidence, it's noise with a schema. Compressed by incremental PCA to 24 components and concatenated with rank-normalised count and time-delta columns, they become a 64-dimensional behaviour vector stored on every Transaction . Now "which confirmed-fraud transactions behave like this one" is a vector query grounded in data, rather than a language model's impression of a table. Mistake one: the closed-case file is not a sample of alerts Here's the part I'd want to read in someone else's write-up. The plan was to measure likelihood ratios instead of guessing them: replay all 5,565 closed investigations through the same extractor the agent uses, under each case's own as_of , and count. Real numbers from real outcomes. The first run produced this: | signal | P(s | fraud) | P(s | cleared) | LR | |---|---|---|---| known_device_and_region | 0.610 | 0.188 | 3.30 | amount_typical | 0.394 | 0.150 | 2.63 | device_new_on_account | 0.134 | 0.484 | 0.28 | amount_anomalous | 0.078 | 0.125 | 0.62 | Read that carefully. It says a transaction on a known device, in a known region, for a typical amount is over three times more likely to be fraud. And that a large, anomalous purchase from a brand-new device is evidence of legitimacy. Which is, taken at face value, insane. And the agent believed it completely - 93% held-out accuracy, beautiful Brier score, and on the benchmark it confidently called quiet, ordinary transactions fraud while clearing the obvious ones. The bug isn't in the modelling. It's in what the closed-case file is. Every cleared case in that history got there by scoring high on the bank's model. It's a travel trip, a new phone, an unusual but genuine purchase - an investigation that happened because the transaction looked strange. Every confirmed case got there because a cardholder phoned up about a charge they didn't recognise, and those are frequently small and unremarkable. So inside the closed-case file, "looks anomalous" genuinely does predict cleared. The measurement was correct. The population was wrong. The fix is a third stratum the file doesn't contain: ordinary transactions from the same four months that no investigation ever touched. Because months 1-4 were fully worked by the bank's analysts, an untouched transaction there is a usable negative - and now the negative class represents non-fraud rather than representing false alarms. Mistake two: unmatched negatives are confounded by card tenure Adding untouched negatives helped, and did not fix it. known_device_and_region came down to LR 1.56 - still pointing the wrong way. The remaining confound is the card, not the transaction. Confirmed-fraud cases sit disproportionately on long-lived, busy cards: more history, therefore more established devices and regions, therefore "the device and region are already established" correlates with fraud through a variable that has nothing to do with fraud. The fix is a matched control design. For each confirmed-fraud case, draw another transaction on the same card as its control. The card-level confound cancels exactly, and the question the calibration answers becomes the question an analyst actually asks: What is different about this transaction compared with the rest of this card's behaviour? And then the uncomfortable part With both corrections in, held-out fraud-versus-not accuracy went from 0.93 to 0.57, and most of the likelihood ratios collapsed to about 1. My first instinct was that I'd broken something. I hadn't. That number is the data answering the question I'd finally asked correctly. Every confirmed case in this history was found by a cardholder ringing up. Fraud never entered that file because it looked anomalous - it entered because someone noticed a charge. So once you remove the trigger as a feature, which you must because it is perfectly confounded with the outcome, the graph signals genuinely cannot separate confirmed fraud from ordinary activity on the same card. Most fraud in this dataset is small and unremarkable. That is the point. The 0.93 was the artefact. The 0.57 is the finding. What the history can teach is which typology a fraud is, and it teaches that strongly: | signal | CNP-new-device | out-of-region | account takeover | |---|---|---|---| device_new_on_account | 0.85 | 0.001 | 0.07 | match_flag_anomaly | 0.001 | 0.56 | 0.47 | online_burst_2_to_4 | 0.43 | 0.001 | 0.04 | known_device_and_region | 0.14 | 0.86 | 0.75 | So the architecture follows the evidence rather than the other way round: fraud-versus-not is carried by the trigger prior and by the st