Stop ranking small samples by percentage: Wilson intervals with real SMS data
We publish a small table of how often SMS verification codes arrive on free virtual numbers, per app and per country. Ranking it by raw success rate is wrong in a way that is easy to miss: a pair with 3 codes in 3 tries (100%) sits above a pair with 45 codes in 46 tries (98%).
Nobody believes 3 of 3 is better evidence than 45 of 46, but a sort by percentage does. This post shows the fix we use, the Wilson score interval, in a few lines of Python and JavaScript, with our real numbers.
The problem with raw rates
Here are four rows from our data (free shared numbers, 27 Oct - 16 Nov 2026):
| App / country | Codes | Tries | Raw rate |
|---|---|---|---|
| YouTube / Cambodia | 6 | 6 | 100% |
| Marktplaats / France | 45 | 46 | 98% |
| YouTube / Pakistan | 13 | 16 | 81% |
| OpenAI / India | 19 | 27 | 70% |
Sorting by raw rate puts the 6-try row first. That is noise ranking above evidence.
Wilson score interval
For k successes out of n tries, the Wilson interval gives a range that the true rate probably lies in (95% with z = 1.96). Unlike the simple p ยฑ 1.96ยทโ(p(1โp)/n) formula, it behaves well near 0% and 100% and for small n.
import math
def wilson(k, n, z=1.96):
if n == 0:
return 0.0, 0.0
p = k / n
d = 1 + z * z / n
c = (p + z * z / (2 * n)) / d
h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return max(0.0, c - h), min(1.0, c + h)
for name, k, n in [("YouTube/KH", 6, 6), ("Marktplaats/FR", 45, 46), ("YouTube/PK", 13, 16), ("OpenAI/IN", 19, 27)]:
lo, hi = wilson(k, n)
print(f"{name:16} {k:>2}/{n:<2}{lo:.0%} - {hi:.0%}")
Output:
YouTube/KH 6/6 61% - 100%
Marktplaats/FR 45/46 89% - 100%
YouTube/PK 13/16 57% - 93%
OpenAI/IN 19/27 52% - 84%
The same in JavaScript, for a front-end table:
const wilson = (k, n, z = 1.96) => {
if (!n) return [0, 0];
const p = k / n, d = 1 + (z * z) / n;
const c = (p + (z * z) / (2 * n)) / d;
const h = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / d;
return [Math.max(0, c - h), Math.min(1, c + h)];
};
Rank by the lower bound
Sort by the lower end of the interval and the ranking matches intuition: Marktplaats/France (89%) first, then YouTube/Cambodia (61%), YouTube/Pakistan (57%) and OpenAI/India (52%). A lucky small sample can still rank well, but only as high as its data supports.
The upper bound is useful too. Telegram on shared numbers had 8 codes in 127 tries; its interval is roughly 3% to 12%. Even the optimistic end is low, so we can say with some confidence that Telegram refuses shared numbers rather than being unlucky.
When intervals overlap, say so
By country, France (55 of 62 tries, about 78%-94%) is clearly ahead, and Mexico (9 of 142, about 3%-12%) is clearly behind. Germany, Italy, Nigeria, India and Pakistan have overlapping intervals spread across roughly 20%-65%, so ranking them against each other would be overclaiming. We show them as one middle group.
Takeaways
- Never sort small-sample rates by the raw percentage.
- Rank by the Wilson lower bound; use the upper bound to call something "reliably bad".
- Show the interval or a "small sample" label next to every rate under ~10 tries.
- If intervals overlap, don't rank: group.
The data is open (CC BY 4.0) on Kaggle, and there is a runnable Colab notebook with the charts.
Disclosure: I work on receive-smss.net, the virtual number service whose free-number logs these figures come from.
Comments
No comments yet. Start the discussion.