Don't ask an LLM for a confidence score

Why I Hate LLM Confidence Scores

Hello, Reader. If someone sent you this post, you have probably tried to extrude a confidence score out of an LLM. I’ve now had this conversation at multiple companies with multiple people, and I’ve had it enough times that it seems like there’s a broader misunderstanding at work here. So my hope is that I can just send people this post instead of relitigating it in a thread every six months.

The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless. I want to be upfront that I’d love to be wrong about this. There are people much, much smarter than me working in this space (and I work with many of them!). If you can rebut any of the arguments below, or point me at research I haven’t read, I would genuinely enjoy that conversation. But absent that, these are my current thoughts as I understand them.

What I Keep Seeing

The pattern shows up everywhere. Chat-based outputs, structured outputs, agentic task results. Someone wants the model to hand back a JSON object with a response key and a confidence key. And more often than not, that confidence key is a continuous score from 0 to 100.

That continuous scale is what really grinds my gears, and I’ll come back to why. But the broader problem is that there’s no scientific validity behind any of it. What you have built is a psychological safety trick. It makes the output feel more trustworthy without making it more trustworthy, and I’d argue the people shipping this experience are mostly lying to themselves about what they’ve shipped.

LLMs Cannot Reliably Quantify Their Own Confidence

I recognize that Anthropic and others have published research arguing that models maintain some kind of latent internal state while they generate. Anthropic’s interpretability work on tracing the thoughts of a language model found that Claude plans several words ahead when writing a rhyming couplet, which is not the behavior you’d predict from a naive next-token story. And their more recent work on emergent introspective awareness found that models can sometimes notice concepts injected into their activations and report on them accurately. That’s real, and it’s interesting.

But read the caveat the researchers themselves put on it: the capability is highly unreliable and highly context-dependent. We do not currently have a strong enough understanding of that internal state to assert that models have any usable ability to assess their own correctness. Noticing that something was injected into your activations under laboratory conditions is a very long way from quantifying how likely your paragraph about a customer’s refund policy is to be right.

If we think about what’s happening under the hood, there is some quasi-cognition going on in the sense of symbolic relationships between aspects of language. With reasoning models, there’s some observation of that process happening too. But this is where the whole thing collapses for me, because it immediately regresses into a who-watches-the-watchmen problem.

Say we want to tap into the thinking trace and use it to evaluate the self-correctness of a response. Fine. That approach has yielded real performance benefits, and I don’t dispute those. But if we’re talking about actually quantifying confidence, the thinking step can’t rescue us, because the obvious next question is: what’s the confidence of the confidence score produced by my thinking step? And then what’s the confidence of that? It’s confidence all the way down.

Do I think reflective reasoning patterns help models catch some of their own errors? Yes. Do I think models currently have a sufficient understanding of their own state to quantify that? No. It’s also worth noting that the research on intrinsic self-correction is not especially encouraging on this front. The DeepMind paper Large Language Models Cannot Self-Correct Reasoning Yet found that when models attempt to correct their initial responses using only their inherent capabilities, without external feedback, performance often degrades.

Confident of What, Exactly?

Here’s the question I now ask immediately whenever someone requests a confidence score: what is your heuristic for confidence? Because “confidence” gets ambiguous extremely fast. Not slightly ambiguous. Extremely.

Is the model confident in the correctness of the response? In the coherence of the response? That it has attempted to fulfill the goal of the user’s request? Those are three completely different questions with three completely different failure modes, and a single float between 0 and 100 flattens all of them into the same number. Absent some heuristic that bounds the concept, these scores are effectively useless.

Confidence Isn’t Uniform Across a Response

There’s another problem with treating this as a single number, and it goes back to internal state. In traditional machine learning, we have real mechanisms for this. A classifier can output a score that at least serves as a legitimate proxy for confidence, and in a regression setting you can put a prediction interval around an estimate.

I want to be careful with that comparison, because classical models don’t hand you a trustworthy number for free either. Guo et al. showed in On Calibration of Modern Neural Networks that modern networks are systematically overconfident, and that raw softmax outputs need post-hoc correction. The difference is that we have an agreed-upon methodology for doing that correction:

• Platt scaling,
• isotonic regression,
• temperature scaling,

all fit against held-out ground truth and checked with reliability diagrams and Brier scores. The number earns trust by being measured against reality, and we have a way to check whether that measurement still holds. That machinery is exactly what’s missing when someone adds a confidence key to a JSON schema.

The closest analog an LLM has natively is the predicted likelihood of each next token. As a side note, I think “LLMs just lossily predict the next token” has become a real oversimplification of where these systems are, especially once you account for reasoning, adaptive thinking, post-training, and the classifiers running on top of the response. But set that aside and take the most simplistic version.

Even there, the problem is that tokens are not uniform. If I’m generating the first hundred words of a response, every single token has its own likelihood value. Some of those are structural glue with near-certain probabilities. Some of them are the load-bearing factual claim in the entire paragraph. Collapsing that distribution into one number that the model then verbalizes is not a measurement. It’s a vibe.

This isn’t just my intuition. It’s the reason an entire research line exists to route around raw token probabilities. Semantic entropy, introduced by Farquhar et al. in Nature, samples a model repeatedly, clusters the responses by meaning rather than by wording, and measures the entropy across those meaning clusters. The reason to go to that trouble is that token-level likelihood conflates “I could have phrased this five different ways” with “I don’t actually know this.”

And note what the method requires: multiple samples and an external comparison step. It’s a measurement performed on the model from the outside, not a number the model reports about itself. I’ve gone looking, and from every piece of research I’ve been able to find, I can’t locate a scenario where these self-reported scores demonstrate the kind of validity people assume they have when they put them in a JSON schema.

If You Really Wanted One, Here’s the Work

I want to be clear that I’m not claiming it’s impossible to get some reliable indication of confidence out of an LLM. I’m claiming that nobody I’ve met wants it badly enough to do the work required. That work would look something like this.

You’d need a strong prompt calibration process in which you define actual heuristics for the model to self-classify against, preferably across a small set of categorical labels rather than a continuous 0-to-100 scale. Then you’d need a lot of calibration and prompt optimization to demonstrate, over some sufficiently large n of responses, that the scores are actually truthy.

The research here is more supportive than my general position might suggest, and I want to represent it fairly. Lin et al. showed in Teaching Models to Express Their Uncertainty in Words that a model can be trained to emit calibrated verbal confidence, though notably they fine-tuned on a purpose-built task distribution to get there. Tian et al. found in Just Ask for Calibration that with the right prompting strategy, RLHF’d models verbalize probabilities that are better calibrated than the model’s own conditional probabilities, and that prompting plus temperature scaling can cut expected calibration error by more than half. And Anthropic’s Language Models (Mostly) Know What They Know found encouraging results asking models to estimate the probability that their own proposed answer is true.

These are the strongest counters to my argument I can find. But if you look at the caveats that come with these approaches (fine-tuning, or a deliberately chosen prompting strategy, or a constrained answer format, or explicit post-hoc scaling), that’s the extra work you have to do that you don’t get out of the box. It is not what happens when you add "confidence": number to a Pydantic model and ship it.

Meanwhile, the argument for categorical over continuous keeps getting stronger. Xiong et al.’s empirical evaluation of confidence elicitation documented how badly overconfident verbalized scores tend to be. And a paper from earlier this year, Rescaling Confidence: What Scale Design Reveals About LLM Metacognition, found something I find genuinely damning: models don’t use the 0-to-100 scale as a continuous spectrum at all. They cluster on round-number anchors. Across six models, more than 78% of responses landed on just three values, with one model reporting exactly 100 on 68% of instances. Coarser scales performed better on metacognitive sensitivity than the 0-to-100 baseline.

So the continuous score that everyone reaches for first is the one with the least support in the literature. If you’re going to do this at all, the label set should be small and the categories should mean something specific that you defined.

Which brings me to a related gripe. LLMs do function decently as ad hoc classifiers, and I’ll grant that. But I think reaching for one is usually a lazy approach. It’s great for labeling synthetic data or for genuinely ad hoc work where you’re not strict on the success criteria. For anything serious in production, I’d much rather train a classical classifier, even if I use an LLM to help bootstrap the training data. And note that if you go the categorical-confidence route, you’re right back to needing humans to define the heuristics behind each label anyway.

Confidence Is Almost Always a Proxy for Correctness

At least the way I’ve seen it used in practice, “confidence” is really a stand-in for correctness, or for certainty. Which makes it another variation of the factuality and grounding problem wearing a different hat.

Case in point: most of the examples I encounter are RAG use cases. Somebody wants the model to indicate whether it’s confident it retrieved the right documents, pulled the data it actually needed, and that its response is accurate to what it pulled. It’s being used as a trust signal.

Here’s where I think that gets genuinely risky, and it’s not the objection people expect. A big part of the value of an LLM in a retrieval system is synthesis. Suppose we perfectly tuned a confidence score where the heuristic was “every explicit fact in this response is grounded in a retrieved document.” Optimizing hard toward that score would reduce the model’s willingness to synthesize across sources.

Yes, ideally you’d end up with a set of verified facts and the model could still draw assertions between them. But I’ve seen enough side effects from prompt optimization that I’d be skeptical of overtuning toward aggressively cited factuality, because a lot of the value is the model inferring across retrieved context and its training data to get somewhere neither one contained on its own.

There’s also a cost problem hiding in that heuristic. If correctness means “cited in an external source,” how do you validate a fact that came from training data? You end up spending retrieval calls on things that are self-evident. I don’t want my LLM making a web search call to find out whether the sky is blue. That’s a waste of my time and a waste of my tokens.

I’d expect a model’s training data to be strong enough that if I asked it to check the claims in its own response, it could handle “the sky is blue” without assistance. Absent some ecological phenomenon, or chemical/biological warfare, that one is settled. Where it gets genuinely hard is specific business logic for a process the model has been asked to operate inside of, and that’s exactly where you start losing the synthesis and the intuition that made the model useful in the first place.

Every Line of a Prompt Is a Liability

I want to take a detour, because this is the part I think people underestimate most. At this point I’ve been working on LLM experiences for several years, and I cannot tell you how many times I’ve watched a team go off the rails by tuning a prompt to correct a specific issue and overcorrecting straight past the target.

One example sticks with me. In one of our support experiences, we had a problem where the LLM wasn’t reliably finding the documents that detailed our escalation process. So we did some prompt optimization against some ad hoc evals to correct it. And we shifted it so far that the model then often found an escalation path immediately, without ever considering what our frontline advisors could resolve themselves. We’d spent months responding to feedb