What Is a P-Value? Worked by Shuffling Nine Real Orders 126 Ways

By Michael Nocito, data analyst · Published August 9, 2026 By the end of this page you can say what a p-value measures in one sentence, compute one by hand with no distribution theory at all, and name the four things people routinely claim a p-value says that it does not. The worked example is nine real orders where two regions differ by 168.50, and the answer comes out of counting rather than out of a table. Here is what to actually do today. Any time you are about to report that two groups differ, write down the two group sizes first. If either is under about twenty, a p-value will almost certainly come back large no matter how real the difference is, and the honest report is the difference, the sizes, and an interval, not a verdict. The short version: a p-value is the share of results at least as extreme as yours that you would get if the thing you are testing had no effect at all. Small means your result would be unusual under nothing-happening. It does not mean the effect is large, and it does not mean the effect is real. That definition is doing a lot of work in one sentence, so it gets the picture. The original carries a diagram here. In words: A histogram built from small dots, one dot per outcome, arranged in ten vertical columns of different heights standing on a horizontal baseline. The columns rise from one dot at the far left to a peak of twenty-seven dots just right of centre, then fall away to a single dot at the far right, giving the whole shape a rounded hump centred slightly left of the middle of the picture. Two vertical dashed lines cut down through the shape, one on the left of the hump and one on the right, placed symmetrically about the hump's centre. The dots lying in the two tails beyond those lines are drawn in a darker, warmer shade, and the dots in the bulk between them are drawn in a lighter blue, so the tails stand out from the middle. In the two columns the dashed lines pass through, the darker dots are stacked at the bottom of the column and the lighter ones above them. A short arrow points down at a spot just inside the right-hand dashed line and is labelled your result. Roughly a third of all the dots in the picture are in the darker shade, and they sit entirely in the two tails. Every number on this page is real. Nine orders from the sixteen-row table used across these guides, with the permutation test run exhaustively rather than sampled. The same table appears in mean vs median and standard deviation. 1. The question a p-value answers Before the definition: you compare two regions and one is ahead by 168.50. Name the specific doubt that number leaves you with, in your own words, before reading on. The doubt is that the gap might be nothing but which orders happened to land where. Split any nine orders into two groups at random and the two group averages will differ, always, by something. So the useful question is not "do they differ" but "is this gap bigger than the gaps random splitting produces?" A p-value answers exactly that question and nothing beyond it. Formally: assuming there is genuinely no difference between the groups, the p-value is the probability of seeing a gap at least as large as the one you saw. Everything else people say about p-values is a paraphrase that has drifted. The assumption in the middle of that sentence has a name, the null hypothesis. It is the boring explanation: the regions are the same, the button colour makes no difference, the drug does nothing. The p-value never evaluates your interesting hypothesis. It only measures how uncomfortable your data would be for the boring one. 2. The data: two regions, 168.50 apart Before the arithmetic: here are nine orders. Look at them and decide, by eye, whether you think South genuinely sells bigger than North, before any statistics. North (5 orders): 880 240 425 440 510 mean 499.00 South (4 orders): 850 660 280 880 mean 667.50 Difference in means: 667.50 − 499.00 = 168.50 South's average order is 168.50 higher, which is a 34 percent gap and would be a real finding if it held up. Look at the raw values though. North has an 880 and South has a 280, and if you swapped just those two rows between the regions the gap would nearly close. Nine numbers is not many, and that intuition is precisely what the next section makes exact. 3. The shuffle test, all 126 of them Before the method: if the region label made no difference at all, then any four of these nine orders could equally well have been the South ones. Say how you would use that fact to test the gap, before reading on. You take the nine order values, forget which region they came from, and deal four of them into a pretend South and five into a pretend North. Compute that split's gap. Do it again for a different split. The gaps you get are what pure chance produces when the label means nothing. There are exactly 126 ways to choose 4 items from 9, so we do not have to sample. We can do every single one. import itertools, numpy as np vals = np.array([880, 240, 425, 440, 510, 850, 660, 280, 880], float) observed = 667.50 - 499.00 # 168.50 gaps = [] for pick in itertools.combinations(range(9), 4): # 126 of them south = vals[list(pick)] north = vals[[i for i in range(9) if i not in pick]] gaps.append(south.mean() - north.mean()) gaps = np.array(gaps) extreme = (np.abs(gaps) >= abs(observed)).sum() print(len(gaps), extreme, extreme / len(gaps)) 126 45 0.35714285714285715 Forty-five of the 126 possible splits produce a gap at least as large as 168.50, in one direction or the other. That is p = 45 ÷ 126 = 0.357 , and it is the picture at the top of this page: the darker dots are those 45. Read the sentence carefully, because it is the whole concept. If the region label meant nothing, we would see a gap this big or bigger about 36 percent of the time. A thing that happens 36 percent of the time by accident is not evidence of anything. It is an ordinary Tuesday. Note what this calculation did not need. No bell curve, no assumption about the shape of order revenue, no degrees of freedom, no table in the back of a textbook. Just counting. This is called a permutation test , and when the group sizes are small enough to enumerate, it is the most honest version of the idea there is. Say out loud why the test uses "at least as large" rather than "exactly this large". The reason is that any exact gap is rare; there are 126 splits and most gaps appear only a handful of times. Asking "how often is chance at least this impressive" is the only version of the question that has a stable answer. 4. The t-test: the same answer without the shuffling Before the comparison: the shuffle needed 126 calculations for nine values. Predict roughly how many it would need for a hundred values per group, and you will see why the classical test exists. Choosing 100 from 200 gives a number with 59 digits in it. Enumeration stops being possible almost immediately, and the classical tests are shortcuts that get to the same answer using mathematics instead of brute force. from scipy import stats stats.ttest_ind(south, north, equal_var=False) # statistic = 0.9709, pvalue = 0.3692 The t-test gives p = 0.369 against the shuffle's 0.357. The two agree, which is the point: the t-test is not a different concept, it is the same question answered by assuming a shape for the data rather than generating it. When that assumption is reasonable, it is faster and it works on any sample size. When the data is heavily skewed or tiny, the shuffle is the one to trust. Two practical notes on the function. equal_var=False asks for Welch's version, which does not assume the two groups have the same spread. Make it your default, because it costs almost nothing when the spreads do match and it is the correct answer when they do not. And the test is two-sided by default, meaning it counts gaps in both directions, which matches the abs() in our shuffle code. Here is the same test in SQL-adjacent form, for when your data lives in a warehouse and you want the ingredients rather than the verdict. SELECT Region, COUNT(*) AS n, ROUND(AVG(Revenue), 2) AS mean_order, ROUND(STDDEV_SAMP(Revenue), 2) AS sd FROM Orders WHERE Region IN ('North', 'South') GROUP BY Region; -- North 5 499.00 235.22 -- South 4 667.50 276.09 5. What 0.05 is, and what it is not Before the section: you have p = 0.36 and the convention says 0.05. Decide what you would write in a report, in one sentence, before reading mine. The 0.05 threshold is a convention, chosen for convenience, with no mathematical claim behind it. Nothing changes in the world between p = 0.049 and p = 0.051. Treating the first as a discovery and the second as nothing is the single most common error in applied statistics, and it is an error of habit rather than of arithmetic. Here are the four things a p-value is regularly claimed to mean, and what it actually means instead. These are worth learning as a set, because each one shows up in real meetings. | The claim | What is wrong with it | |---|---| | "p = 0.36, so there is a 36 percent chance the regions are the same" | Backwards. The p-value assumes they are the same and asks about the data. It cannot tell you the probability of the assumption. | | "p = 0.03, so the effect is important" | Different question. A tiny, useless difference gets a small p on enough rows. Size and significance are separate. | | "p = 0.36, so there is no difference" | Not tested. Failing to detect a difference on nine orders is not evidence that none exists. | | "p = 0.04, so it will replicate" | Not implied. A p-value describes this one data set under one assumption. Repeat the study and it can land anywhere. | What I would actually write for our example: "South's average order was 168.50 higher than North's, on 4 orders against 5. A gap that size arises by chance about a third of the time at these sample sizes, so this data cannot tell us whether the regions really differ. The 95 percent interval on the gap runs from −256 to +593." That sentence has th