The Explanation Gap: Why Explainable AI Still Struggles to Speak Human

The Explanation Gap

A model predicts a 92% risk of hospital readmission. The SHAP values show age, prior admissions, and medication adherence as the top contributors. But when a nurse asks, “Why this patient, this week?” the technical answer is: “coefficients multiplied by feature values, printed to four decimal places.” Well, this is correct as it is but almost useless in this conversation. This is the unresolved core problem of explainable AI: getting a model to compute an answer was never the hard part in the long run. Getting it to explain that answer to the person who actually has to act on it, without making the explanation sound more certain than it deserves to be, is the part the field is still working through.

Why Explainability Isn't Optional

In 2015, DARPA launched its Explainable Artificial Intelligence (XAI) program with an explicit goal: to enable end users to “understand, appropriately trust, and effectively manage” AI systems. The program's retrospective makes clear that this was not an academic exercise. High‑stakes domains (healthcare, criminal justice, credit, hiring) cannot tolerate decisions that are accurate on aggregate but inscrutable in individual cases.

The field has since developed a rich landscape of explanation methods:

• LIME (Local Interpretable Model‑agnostic Explanations) fits simple surrogate models around individual predictions.
• Counterfactual explanations answer “what would need to change for the outcome to flip?”
• Saliency maps highlight influential pixels in images.
• Attention‑based explanations point to tokens a transformer model “attended to.”
• Rule extraction methods distill complex models into human‑readable if‑then statements.

None of these methods, on their own, answers the question that matters most in high‑stakes decisions: Why should I, as a clinician or regulator or affected individual, believe this explanation? Raw accuracy metrics “the model is 92% accurate” do not address this. They describe aggregate performance, not individual reasoning. A model can be highly accurate and still rely on spurious correlations, proxy variables for protected attributes, or patterns that will not hold under intervention. In healthcare, where a single false negative can cost a life, the “why” is not optional. It is the entire point.

The European Data Protection Supervisor's 2023 TechDispatch on XAI puts it plainly: “It is therefore unacceptable to have a ‘black box’ effect that hides the underlying logic of decisions made by AI.” Opacity can hide bias, inaccuracies, and hallucinations. It prevents affected individuals from understanding, challenging, or correcting decisions that shape their lives.

What SHAP Actual Is (...and Isn't)

To understand why the communication problem is so hard, you need to understand what SHAP values actually represent. SHAP (SHapley Additive exPlanations) is a method for attributing a model's prediction for a single instance to its input features. It is grounded in Shapley values from cooperative game theory, which answer: given a payoff produced by a coalition of players, how much credit does each player deserve?

In the ML context: the “players” are input features. The “payoff” is the model's prediction for a specific instance. The Shapley value for feature i is the average marginal contribution of that feature across all possible coalitions of other features. Formally, for a feature i and a set of features F, the Shapley value is:

ϕ_i = Σ_{S ⊆ F \ i} |F|! / (|S|! (|F| - |S| - 1)!) [ f(S ∪ i) - f(S) ]

where f(S) is the model's prediction when only features in S are known.

I know, I know. The formula looks intimidating, right? My bad. Let me explain: Imagine you're trying to figure out why your friend got 10/10 on a test. You know they studied three things: Studying, Getting enough sleep, Practising questions. SHAP is like asking: “How much did each of these things help my friend get that 10/10?” But there's a catch: studying might help a lot when combined with practice, but not as much without it. So SHAP tries different combinations:

• Studying alone → score goes up a little
• Studying + practice → score goes up a lot
• Studying + sleep + practice → score goes up even more

It looks at all these different combinations and works out the average contribution of studying. That's what a Shapley value is: a way of fairly figuring out how much each feature helped produce the final prediction.

SHAP satisfies three desirable properties:

1. Local accuracy - the attributions sum to the prediction.
2. Missingness - features not present get zero attribution.
3. Consistency - if a feature's contribution increases, its attribution does not decrease.

What SHAP is not: a causal explanation. This distinction is easy to lose the moment attributions get turned into sentences. A SHAP value of +0.18 for “prior admissions” means: knowing this patient's prior admissions pushes the model's prediction 18 percentage points above the baseline. It does not mean that reducing prior admissions would lower risk by 18 points in the real world. As the SHAP documentation warns: “SHAP makes transparent the correlations picked up by predictive ML models. But making correlations transparent does not make them causal!”

If the model has learned that zip code is a proxy for race in a biased dataset, SHAP will dutifully explain that spurious association. This is not a bug in SHAP. It is a feature of any method that explains model behavior rather than data‑generating processes. The moment you translate SHAP values into prose; “this factor drove the prediction,” “this variable caused the risk”; you risk smuggling in causal claims the method does not support.

The Communication Gap

Here is the crux of the problem. A typical SHAP output for a single prediction looks like this (simplified):

Base value: 0.12
shap_values: [0.08, -0.03, 0.15, 0.02, -0.01]
feature_names: ['age', 'income', 'prior_admissions', 'medication_adherence', 'distance_to_clinic']

Or, in a more structured form:

Explanation (
    base_values = 0.12,
    values = [0.08, -0.03, 0.15, 0.02, -0.01],
    feature_names = [' age ', ' income ', ' prior_admissions ', ' medication_adherence ', ' distance_to_clinic ']
)

For an ML engineer, this is informative as he/she reads Age: +0.08, Income: -0.03 fluently. For a nurse, a patient, a regulator, or a loan officer, it is nearly useless. The numbers live on a transformed scale (often log‑odds). The feature names are internal identifiers, not clinical or business terms. A patient, a case worker, or a manager generally cannot, and wouldn't know how much weight to give it even if they could parse the syntax.

There is no context: is 0.15 “large” for prior admissions, or typical? Christoph Molnar's Interpretable Machine Learning emphasizes this repeatedly: interpretability is not just about producing an explanation, but producing one that the intended audience can comprehend and use. A data scientist needs