Choosing a Decision Threshold With Real Costs

A classifier outputs a probability; a system needs a decision. The conversion is a threshold, and 0.5 is only optimal when a false positive and a false negative cost exactly the same. They almost never do. Given four numbers, the right threshold is one line of algebra - here it comes out at 0.258, and using 0.5 instead costs £3,570 per 10,000 transactions. 0.5 is an assumption, not a default predict() in scikit-learn thresholds at 0.5. That is a reasonable library default and a poor production decision, because it encodes a claim: that being wrong in one direction costs precisely as much as being wrong in the other. In fraud, a missed fraud costs a chargeback and a blocked customer costs a sale. In medical triage the two are not remotely comparable. In content moderation they are different kinds of harm entirely. Two preconditions before any of the arithmetic below is meaningful. The scores must be calibrated probabilities, because the threshold that comes out is on the probability scale - an uncalibrated 0.258 is not 0.258. And the four costs must be real numbers agreed with whoever owns the budget, not guesses supplied by the modeller. The cost matrix A payment fraud screen. Four outcomes, four costs, in pounds per transaction. These are the numbers to argue about; everything after is arithmetic. | Outcome | Description | |---|---| | true positive | Fraud, correctly held. Cost £5 - the manual review that clears or confirms it. Not zero: acting on a correct flag still costs something, and that is the number teams most often forget. | | false positive | Legitimate payment, wrongly held. Cost £40 - the lost margin on an abandoned order, the support contact, and a share of the increased churn from a customer who was accused of fraud. | | false negative | Fraud, allowed through. Cost £120 - the chargeback, the scheme fee, and the goods that shipped. | | true negative | Legitimate payment, allowed. Cost £0. The baseline everything else is measured against. | Getting these numbers is the hard part of the exercise and it is usually a two-hour conversation, not a research project. Finance knows the chargeback cost. Support knows the contact cost. The margin is in the P&L. Where a number is genuinely unknown, put a range on it and check whether the resulting threshold moves enough to matter. The threshold, derived For a transaction with fraud probability p, compare the expected cost of each action. E[cost | flag] = p * C_TP + (1 - p) * C_FP E[cost | pass] = p * C_FN + (1 - p) * C_TN Flag when flagging is cheaper. The break-even p is where they are equal: pC_TP + (1-p)C_FP = pC_FN + (1-p)C_TN Rearranged: p = (C_FP - C_TN) / [ (C_FP - C_TN) + (C_FN - C_TP) ] Substituting 5, 40, 120, 0: p = (40 - 0) / [ (40 - 0) + (120 - 5) ] = 40 / (40 + 115) = 40 / 155 = 0.258 Flag anything scoring above 0.258. The formula has the right limits: if a false positive were free the threshold goes to 0 (flag everything); if a false negative were free it goes to 1 (flag nothing); and it returns 0.5 exactly when the two error costs are equal, which is the special case the library default assumes. Note that the base rate does not appear. It does not need to - the model’s calibrated probability already carries it. A rare-event problem does not need a different formula; it needs a calibrated model. Verified against the score distribution The per-transaction rule is optimal transaction by transaction, so it must also minimise the total. Here is that check on 10,000 scored transactions grouped into six score bands. Invented distribution, arithmetic checkable. band mean p n cost if ALL flagged cost if ALL passed n*[p5 + (1-p)40] n[p120] 0.00-0.01 0.003 8,000 319,160 2,880 pass 0.01-0.05 0.030 1,200 46,740 4,320 pass 0.05-0.15 0.100 400 14,600 4,800 pass 0.15-0.30 0.220 180 5,814 4,752 pass 0.30-0.60 0.450 120 2,910 6,480 FLAG 0.60-1.00 0.800 100 1,200 9,600 FLAG Expected frauds = 8000(.003) + 1200(.03) + 400(.10) + 180(.22) + 120(.45) + 100(.80) = 24 + 36 + 40 + 39.6 + 54 + 80 = 273.6 (2.74%) TOTAL COST BY POLICY threshold 0.258 (flag the last two bands) 2,880 + 4,320 + 4,800 + 4,752 + 2,910 + 1,200 = 20,862 threshold 0.50 (flag only the top band) 2,880 + 4,320 + 4,800 + 4,752 + 6,480 + 1,200 = 24,432 threshold 0.15 (flag the last three) 2,880 + 4,320 + 4,800 + 5,814 + 2,910 + 1,200 = 21,924 threshold 0.05 (flag the last four) 2,880 + 4,320 + 14,600 + 5,814 + 2,910 + 1,200 = 31,724 The derived threshold wins, as it must. cost of using 0.5 instead: 24,432 - 20,862 = £3,570 per 10,000 = 14.6% of total cost The 0.15 row is worth noticing. It is close to optimal - £1,062 worse - because the 0.15-0.30 band straddles the break-even point and the cost curve is flat near its minimum. That flatness is good news: you do not need the threshold to three decimal places, and a threshold set from approximate costs will capture most of the available gain. When the costs are ranges, not numbers The usual objection to this whole exercise is that nobody knows the four costs precisely. That objection is answerable, and answering it is more persuasive than the point estimate was. Take the widest defensible range for each: a false negative somewhere between £90 and £150, a false positive between £30 and £50, review at £5. Evaluate p* at both corners. p* = C_FP / (C_FP + C_FN - C_TP) Most aggressive case (cheap false positive, expensive miss): C_FP = 30, C_FN = 150 p* = 30 / (30 + 150 - 5) = 30 / 175 = 0.171 Most conservative case (expensive false positive, cheap miss): C_FP = 50, C_FN = 90 p* = 50 / (50 + 90 - 5) = 50 / 135 = 0.370 So p* lies somewhere in 0.171 to 0.370. 0.5 is outside that entire interval. Under every combination of the costs anybody in the room is willing to defend, the library default is too high. What the uncertainty costs, on the same score distribution: threshold 0.171 (flag the last three bands) £21,924 threshold 0.258 (the midpoint estimate) £20,862 threshold 0.370 (flag the last two bands) £20,862 -------- spread across the whole plausible range £1,062 (5.1%) versus the cost of using 0.5 £3,570 (14.6%) Being uncertain about the costs is three times cheaper than being confidently wrong about them. That is the argument to make when somebody says the numbers are too soft to act on: the softness is worth five per cent, and the default is worth fifteen. It also tells you where to spend effort. If the range on p* had straddled a band boundary with a large population behind it, narrowing the cost estimate would have been worth doing. Here it does not, so the right move is to pick the midpoint, write down the assumption, and revisit it when the margin changes. Why not just maximise F1 Because F1 embeds a cost assumption too, and it is not yours. The F1 score weights precision and recall equally, which corresponds to a particular implicit ratio between false positives and false negatives - one that has nothing to do with £40 and £120. Tuning a threshold to maximise F1 optimises somebody else’s business. The same objection applies to Youden’s J, to the point on the ROC curve nearest the top-left corner, and to every other geometric rule for picking an operating point. They are all cost assumptions in disguise. If you genuinely cannot obtain the costs, say so explicitly and pick a constraint instead - “the highest recall we can reach at 95% precision” or “the top 200 cases per day, because that is what the review team can process”. A capacity constraint is at least an honest statement about the world. The capacity version is common and worth stating properly: sort by score, take the top k where k is what the team can handle, and the threshold is whatever score sits at position k. Then recompute it weekly, because the score distribution moves. Choosing it in code import numpy as np C_TP, C_FP, C_FN, C_TN = 5.0, 40.0, 120.0, 0.0 # 1. the closed form, valid when probabilities are calibrated p_star = (C_FP - C_TN) / ((C_FP - C_TN) + (C_FN - C_TP)) print(f"analytic threshold: {p_star:.4f}") # 2. the empirical sweep, valid always -- and the one to ship, # because it also reports what the choice is worth def total_cost(y_true, p, t): pred = p >= t tp = int(((pred == 1) & (y_true == 1)).sum()) fp = int(((pred == 1) & (y_true == 0)).sum()) fn = int(((pred == 0) & (y_true == 1)).sum()) tn = int(((pred == 0) & (y_true == 0)).sum()) return tp * C_TP + fp * C_FP + fn * C_FN + tn * C_TN grid = np.linspace(0.01, 0.99, 197) costs = np.array([total_cost(y_val, p_val, t) for t in grid]) best = grid[costs.argmin()] print(f"empirical threshold: {best:.4f}") print(f"cost at best: {costs.min():,.0f}") print(f"cost at 0.50: {total_cost(y_val, p_val, 0.50):,.0f}") # how flat is the minimum? the range of thresholds within 1% of optimal tol = grid[costs <= costs.min() * 1.01] print(f"within 1% of optimal: {tol.min():.3f} to {tol.max():.3f}") Run both. The analytic value tells you what the costs imply; the empirical sweep tells you what your scores actually do. A large gap between them is a calibration problem, not a threshold problem, and the fix is upstream. Print the flat region as the last line does - it is the number that tells you how much precision the decision deserves. Thresholds go stale - When the score distribution shifts, a fixed threshold changes what it does. Same threshold, different flag rate, different cost - with no code change and no alert. Monitoring the rate of threshold crossings catches this and is one series. - When you retrain, the threshold does not carry over. A new model has a new score distribution. Re-deriving the threshold on the new model’s validation predictions is part of the retraining job, not a follow-up ticket. - When costs change, re-derive rather than nudge. A change in margin, in chargeback fees or in review-team cost moves p* immediately, and the arithmetic takes a minute. - Different segments can justify different thresholds. If the cost of a false positive is genuinely higher for enterprise customers, they get a higher thresho