The Model Validation Playbook for GenAI: Lessons from Banking

The Model Validation Playbook for GenAI: Lessons from Banking How model validation standards are changing for LLM-based systems: what breaks, what carries over, and how to test output quality Introduction Let's start with a recent, increasingly common scenario in the Risk Management department of large banks. Let's say a risk model validator at a large bank opens a submission. The model is an AI assistant that reads a borrower's financial statements, pulls relevant third-party research, and drafts the first version of a credit memo. It saves analysts several hours a week, and obviously the business wants this AI model to go live next quarter. She opens the standard validation template to start the review process. This template has been refined over a decade of regulatory examinations and worked on every scorecard, every loss forecasting model, every pricing engine she has reviewed. She reaches the first question: "Provide the development sample." But there is no development sample. This gen AI model was trained on a corpus nobody at the bank has seen and by a vendor who won't describe it. And that is only the first question from the remaining ninety. Model risk management was never designed for generative AI in banking. So, this is roughly where every model risk function in the banking/financial industry currently finds itself. An effective challenge on a model we cannot retrain, whose training data we cannot see, looks different from an effective challenge on a traditional scorecard. The craft shifts from replication to test design. Why this framework matters beyond banking The core challenge described in this article, i.e., how to validate a system you cannot fully inspect, is now a problem for anyone deploying generative AI in a serious production context. Whether you are building a medical summarisation tool, a legal research assistant, or a customer-facing chatbot, the same questions apply: What does 'good' look like when there is no ground truth? How do you catch confident errors before they reach a user? The framework that follows in this article, based on risk tiering, outcome-based evaluation, robustness testing, and monitoring for silent drift, was built for banks, but it is directly transferable to any domain where the cost of being wrong matters more than the cost of being slow. What model risk management in banking actually does If you work in data science outside banking, this discipline may be unfamiliar. So let's set up the context properly. Banks run on mostly traditional statistical predictive models. These models decide who gets credit and at what price. Models set how much capital the institution must hold against its loan book. Models forecast losses under hypothetical recessions, value illiquid positions, flag suspicious transactions, and determine reserves that flow directly into published financial statements. When one of these is wrong, the consequences are not an unhappy user; they are mispriced risk, understated reserves, regulatory findings/penalty, and occasionally a very large loss. The industry learned this expensively. Credit models that assumed house prices don't fall nationally contributed materially to the 2008 crisis. A revised risk model at one bank in 2012 understated exposure so badly that a trading loss ran into billions before anyone caught it. Regulators responded by formalising the discipline: US supervisory guidance issued in 2011 (known to everyone in the field as SR 11-7) defined model risk as the potential for adverse consequences from decisions based on incorrect or misused model output, and required banks to manage it deliberately. The EU AI Act codifies a similar expectation for high-risk AI systems used in creditworthiness assessments, pricing, or essential banking services. Its core obligations on risk management, data governance, technical documentation, record-keeping, transparency, human oversight, and accuracy/robustness map closely onto SR 11-7’s conceptual soundness, outcomes analysis, and ongoing monitoring. For global banks, one validation framework can be structured to satisfy both regimes, but the AI Act adds explicit requirements around fundamental rights impact assessments and post-market monitoring that extend the second line’s traditional scope. The model risk management structure is remarkably consistent across large institutions: Line of defence | Who | Role | |---|---|---| First | Business and model development | Builds the model, tests it, owns its performance and its use | Second | Model risk management/validation | Independently challenges the model before approval, and keeps challenging it | Third | Internal audit | Checks that the first two are doing their jobs | The second line is the part this article is about. A validator doesn't just check arithmetic. They ask whether the modelling approach was conceptually appropriate, whether the data supported it, whether the output actually performs, whether the production implementation matches what was approved, and whether the people using the output understand its limits. Nothing goes live without their sign-off, and everything gets re-examined periodically. Three things anchor that review, and they have been stable for over a decade: conceptual soundness (is the approach defensible?), outcomes analysis (does the output hold up when tested?), and ongoing monitoring (is it still working now?). Why Generative AI Breaks Traditional Model Validation Generative AI has arrived in banks faster than any modelling technology in recent memory, and not in a specific shape. It can be complaint summarisation, policy lookup, research retrieval, first drafts of credit memos, internal documentation, literally anything. These models are attractive because they directly influence cost, but they can also be risky. Because they sit close to customers and close to credit decisions. These are exactly the places where a regulated institution has the least appetite for a wrong answer. And the validation apparatus that existed to prevent this risk no longer fits. Every question on the template assumes properties these systems don't have. 1. Five Structural Breaks in Gen AI Model Validation Property | Why classical validation can't absorb it | |---|---| There is no model; there is a system | Query handling, retrieval, prompt template, base model, decoding settings, guardrails, orchestration. Change any one and behaviour shifts | The output is a distribution | Non-determinism is a design feature. Even at the lowest temperature setting, batching and context effects produce variation | No ground truth for open-ended tasks | AUC, KS, Gini and MSE, etc. work on an observable target. Summarisation and drafting have none | You didn't build the core component | No training data, no methodology document, no replication. A system card and benchmark scores on unrelated tasks | Version drift without a change request | Hosted models are updated behind a stable endpoint. Nothing in the change process fires, because procedurally nothing changed | The first row has a direct procedural consequence. The unit of validation is the system, not the model. Two use cases sitting on the same base model are two separate validations. The second row inverts an instinct worth naming. A system that answers correctly 95% of the time and produces a confidently wrong answer 5% of the time is a different object from a model with a 5% error rate on a known error distribution. Fluency is uncorrelated with correctness in a way no calibrated PD model has ever been. Because these structural breaks make blanket validation impossible, risk tiering becomes the lever that determines how much evidence is enough. 2. Risk Tiering for Generative AI: A Practical Framework Everything downstream scales off the risk tiering, so this is where a validator focuses the most. The classical drivers still apply - business reliance, impact of error, complexity, control feasibility. Two more matter specifically here. How far does the output travel? Exposure level | Example | Practical implication | |---|---|---| Input to an internal decision | Summarised research feeding a credit view | The human review step is the control | Reaches a customer or regulator | Correspondence, disclosures, complaint responses | Reputational and conduct risk dominate the model risk | Does it just produce text, or does it also act? A system that drafts something a person reads has low risk. A system that calls tools, writes to systems of record or triggers downstream processes has an entirely different risk profile. 3. The three questions an AI Model Risk Assessment report answers The questions are the same ones we have always asked. What counts as evidence has changed completely. 4. Before we look at the output A. Should this be a language model at all? The cheapest and most valuable challenge available to the second line. A surprising number of proposed use cases are deterministic problems where a generative model introduces variance to solve something a parser handles exactly. If a simpler method gets 90% of the value with none of the fabrication, that comparison belongs in the report. Complexity has to earn its place. B. What exactly is the system? Ask for a component map. Then ask which components the institution actually controls. Component | Who controls it | What it can break | |---|---|---| Retrieval and index | You | Stale or missing evidence; over-broad access | Prompt template | You | Ambiguous instructions; injection exposure | Base model | Vendor | Everything | Decoding settings | You | Output dispersion | Guardrails | You | Both over-blocking and under-blocking | Orchestration | You | Compounding errors across steps | In production, retrieval quality and prompt construction usually dominate performance. The base model is often the least important source of variance, which is fortunate, because it's the part you can't inspect. C. What went into it? The specification is not "we used a