I blocked accessibility overlays on 56 Shopify stores and measured what changed

Median change to the underlying markup: zero - in every one of six runs, across three samples drawn at different times. That sentence needs about two thousand words of context before it means anything, including the three occasions where I nearly published a number that was wrong. Here they are. Start with what it does not measure: whether a widget helps anyone. It measures whether a page's markup changes when the widget runs. A tool with a genuinely useful toolbar - bigger text, a reading mode, a contrast switch - scores zero here and may still be worth having. The full limitations are below, and they are load-bearing. The comparison that cannot work Accessibility overlay widgets attach a script to your site and are sold as a way to improve its accessibility. The obvious way to test that is to measure sites that have one against sites that don't. I did that first. It produced a clean, statistically significant result pointing in an interesting direction, and it is worthless. In a pool of 233 German Shopify storefronts, the 17 stores with an overlay installed had a median of 25 violation nodes; the 216 without had 14. The overlay group carried 78.6% more, and the difference clears the usual bar (Mann-Whitney, z = 2.48, p = 0.013). Read that as a fact about which stores buy a widget, not about what a widget does. Group A is 17 stores and a median over 17 is fragile; the international contour, with 65 stores against 71, shows no statistically detectable difference (5.7%, z = −0.90, p = 0.369). Nor can I say why the German groups differ. Overlay stores do carry heavier pages, a median of 3,068 DOM nodes against 2,714, but that is 13% more markup against 78.6% more violations, so page weight accounts for a small part of the gap at best. The conclusion of this article rests on the paired test in the next section, not on either of these numbers. The problem is structural. Nobody installs an accessibility widget at random. You go looking for one because you have a problem, or because a lawyer's letter arrived, or because your category is one where this gets noticed. The stores that bought a widget are not a control group for the stores that didn't - they are the stores that had a reason. That comparison can never separate the effect of the tool from the reason someone reached for it, no matter how many sites you throw at it or how small the p-value gets. I could have published the 78.6%. It is a real number, it is significant, and it flatters a thesis people enjoy. It is also uninterpretable, and dressing it up with a p-value would only have made it more convincing than it deserves. The comparison that does work Measure the same page twice, a minute apart, and change exactly one thing: whether the browser is allowed to load the overlay's script. ON → load the page as served, wait, run axe-core OFF → load the page again, abort every request to the overlay's CDN, wait, run axe-core Same URL. Same viewport, 1440×900. Same locale and Accept-Language . Same settle delay. Same axe-core version and the same rule set. Same network position - a German exit node throughout - seconds apart, so the same CDN edge and the same storefront state. Fresh browser context each time, so nothing carries over. The difference between those two numbers is what the widget does to the page. There is no selection left to worry about, because the store is its own control. I ran this on 56 distinct stores - a German sample of 16, an international sample of 20, and a fresh sample of 20 that had never been measured, drawn deliberately after the first results existed so that the finding had a chance to fail against data it wasn't derived from. Two of the three samples were measured twice, and one vendor had to be measured again after a bug described below, which is how 56 stores produce 92 store-runs and 179 page pairs. What came out Nodes removed by the widget, per run. Read "removed" as OFF minus ON: positive means the widget took violations away. | run | pairs | stores | nodes removed | of | share | median per store | |---|---|---|---|---|---|---| | German | 14 | 7 | 12 | 468 | 2.6% | 0.0% | | English, run 1 | 35 | 18 | 106 | 1,920 | 5.5% | 0.0% | | English, run 2 | 35 | 18 | 99 | 1,921 | 5.2% | 0.0% | | Fresh, run 1 | 37 | 19 | 41 | 1,343 | 3.1% | 0.0% | | Fresh, run 2 | 35 | 18 | 17 | 1,215 | 1.4% | 0.0% | | Accessibly, re-measured | 23 | 12 | −3 | 905 | −0.3% | 0.0% | | all six | 179 | 56 | 272 | 7,772 | 3.5% | 0.0% | The widgets in the sample were UserWay (23 stores), accessiBe (15), Accessibly (12), EqualWeb (4) and AudioEye (2). Results are not broken down by product: the split is uneven and no per-vendor number here would survive its own sample size. Nothing in these figures should be attached to any one named product. The Accessibly row exists only because that vendor had to be re-measured, not to single it out. Two summary figures, and they disagree on purpose. 3.5% is node-weighted: every violation node counts equally, so one large storefront with hundreds of findings can carry a whole run. 0.0% is the median across stores: every store counts once, regardless of size. The median came out at zero in all six runs. The rest of the shape: - 56 of 92 store-run observations showed no change whatsoever - 61%. - Per-run share ranged from −0.3% to 5.5%. That spread, on samples this size, is the honest measure of how much a single run can wander. - In every run some stores measured worse with the widget enabled than with it blocked - 1, 3, 2, 2, 4 and 2 stores respectively. That is mostly run-to-run noise rather than a widget effect. Across the two English runs, 3 different stores came out worse and only 2 did so both times; across the two fresh runs, 4 different stores and again only 2 both times. In one sentence: on most of these storefronts, blocking the widget's script changed nothing an automated checker could see, and where something did change it was small and did not reproduce reliably between runs on the same stores. The three times I nearly published something false All three were caught by re-running measurements I had already written down as results. All three push the number in the flattering direction. If you repeat this work, you will hit them. Lazy loading An early run showed one German storefront dropping from 46 violation nodes to 21 - a 54% improvement, exactly the kind of number that ends up on a slide. It was not the widget. The two measurements used different settle delays, 5 seconds and 20 seconds. Almost the entire difference is a single rule: image-alt fell from 31 to 7, while the page kept growing between the two measurements. The storefront's own lazy-loading was still working at 5 seconds, and the placeholder images it had not yet swapped out were being counted as images with no alt text. What I had measured was how long I was willing to wait. Re-run as an actual pair - widget allowed and widget blocked, both at the same delay: 46 and 46. This is why the settle delay is not a detail you tune for convenience. Any asymmetry in how long you wait becomes a measurement of page weight wearing an accessibility result's clothes. Pages that never rendered The fresh sample nearly ended the project. Run 1 gave −17 nodes; run 2, on the identical sample under identical settings, gave +116. A result that flips sign between consecutive runs is not a result. Per store, though - before any exclusion - 16 of 20 were stable within three nodes, and three stores produced 92% of the variance. One of them returned a single violation node on the widget-enabled side against 56 on the blocked side: the page had not rendered at all, and it had no . That one blank page contributed +98 of that run's +116. So I wrote down a validity rule - before checking which stores it would remove: A pair is discarded if one side has ≤ 3 violation nodes while the other has ≥ 15, or if the two sides disagree about whether the page had a . It removes 4 pairs across the whole study. With it applied, the fresh sample gave 3.1% and 1.4% - inside the range the original samples had already produced. The finding survived contact with data it had not been built on. I mention the order - rule first, then look - because doing it the other way round is the whole game. A threshold chosen after you have seen which points it excludes is not a threshold, it is an opinion with arithmetic attached. A vendor that was never actually switched off This is the one that should have killed the study, and the only reason it did not is that I checked a field I had been collecting and never using. The OFF side works by aborting every request whose URL contains one of a list of overlay hosts. My list contained accessiblyapp.com . Accessibly serves its script from cdn.accessibly.app . Those two strings do not match, so for every pair of every run, that vendor's widget loaded normally on both sides. Thirty pairs that I was counting as ablations were the same page measured twice. Nothing looked wrong. Those pairs produced small differences clustered around zero - which is exactly what the study concluded - so the bug was holding the headline up rather than knocking it down. A result that agrees with you is the hardest kind to audit. What exposed it: ablation.mjs records widgetInDom on the ON side, whether the widget's own button or panel is present after the delay. Comparing that field between ON and OFF, Accessibly never disappeared - 30 pairs out of 30. Then a second surprise: after fixing the host list, it still never disappeared, because accessibly-trigger and accessibly-config are rendered into the page by the Shopify app itself and sit there whether the script loads or not. widgetInDom is a vendor-dependent proxy and cannot be used as a validity filter - a conclusion I reached only after briefly believing the opposite. What settles it is the request log: with the corrected host list the script request is aborted, and that is the thing being manipulated. The 30 bad pairs are excluded and