DEV Community

Stop ranking small samples by percentage: Wilson intervals with real SMS data

We publish a small table of how often SMS verification codes arrive on free virtual numbers, per app and per country. Ranking it by raw success rate is wrong in a way that is easy to miss: a pair with 3 codes in 3 tries (100%) sits above a pair with 45 codes in 46 tries (98%).

Nobody believes 3 of 3 is better evidence than 45 of 46, but a sort by percentage does. This post shows the fix we use, the Wilson score interval, in a few lines of Python and JavaScript, with our real numbers.

The problem with raw rates

Here are four rows from our data (free shared numbers, 27 Oct - 16 Nov 2026):

App / country Codes Tries Raw rate
YouTube / Cambodia 6 6 100%
Marktplaats / France 45 46 98%
YouTube / Pakistan 13 16 81%
OpenAI / India 19 27 70%

Sorting by raw rate puts the 6-try row first. That is noise ranking above evidence.

Wilson score interval

For k successes out of n tries, the Wilson interval gives a range that the true rate probably lies in (95% with z = 1.96). Unlike the simple p ยฑ 1.96ยทโˆš(p(1โˆ’p)/n) formula, it behaves well near 0% and 100% and for small n.

import math

def wilson(k, n, z=1.96):
    if n == 0:
        return 0.0, 0.0
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return max(0.0, c - h), min(1.0, c + h)

for name, k, n in [("YouTube/KH", 6, 6), ("Marktplaats/FR", 45, 46), ("YouTube/PK", 13, 16), ("OpenAI/IN", 19, 27)]:
    lo, hi = wilson(k, n)
    print(f"{name:16} {k:>2}/{n:<2}{lo:.0%} - {hi:.0%}")

Output:

YouTube/KH       6/6  61% - 100%
Marktplaats/FR  45/46 89% - 100%
YouTube/PK      13/16 57% - 93%
OpenAI/IN       19/27 52% - 84%

The same in JavaScript, for a front-end table:

const wilson = (k, n, z = 1.96) => {
  if (!n) return [0, 0];
  const p = k / n, d = 1 + (z * z) / n;
  const c = (p + (z * z) / (2 * n)) / d;
  const h = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / d;
  return [Math.max(0, c - h), Math.min(1, c + h)];
};

Rank by the lower bound

Sort by the lower end of the interval and the ranking matches intuition: Marktplaats/France (89%) first, then YouTube/Cambodia (61%), YouTube/Pakistan (57%) and OpenAI/India (52%). A lucky small sample can still rank well, but only as high as its data supports.

The upper bound is useful too. Telegram on shared numbers had 8 codes in 127 tries; its interval is roughly 3% to 12%. Even the optimistic end is low, so we can say with some confidence that Telegram refuses shared numbers rather than being unlucky.

When intervals overlap, say so

By country, France (55 of 62 tries, about 78%-94%) is clearly ahead, and Mexico (9 of 142, about 3%-12%) is clearly behind. Germany, Italy, Nigeria, India and Pakistan have overlapping intervals spread across roughly 20%-65%, so ranking them against each other would be overclaiming. We show them as one middle group.

Takeaways

  • Never sort small-sample rates by the raw percentage.
  • Rank by the Wilson lower bound; use the upper bound to call something "reliably bad".
  • Show the interval or a "small sample" label next to every rate under ~10 tries.
  • If intervals overlap, don't rank: group.

The data is open (CC BY 4.0) on Kaggle, and there is a runnable Colab notebook with the charts.

Disclosure: I work on receive-smss.net, the virtual number service whose free-number logs these figures come from.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.