How to find every Shopify store in a list of 10,000 domains

You have a list of domains. A conference attendee export, a directory scrape, a spreadsheet from a colleague. You want to know which of them run Shopify, which run WordPress, which have HubSpot installed, and which are on Cloudflare. That single question sits behind a lot of sales prospecting, competitor tracking and agency audits. The mechanics are not complicated, and it is worth understanding them before you decide whether to build or buy. What a website tells you without being asked Every page you request hands over more than the HTML you asked for: - Response headers. Server: cloudflare and acf-ray header are Cloudflare.X-Powered-By: PHP/8.2 is self-explanatory. Shopify sendsx-shopid . - Cookies. _shopify_y ,__utma ,wp-settings-* each pin a platform. - HTML and meta tags. is about as direct as it gets. - Script sources. cdn.shopify.com ,googletagmanager.com/gtm.js ,js.hs-scripts.com . - DNS records. MX records name the mail provider. TXT records carry verification tokens for every SaaS tool the company has ever set up, which is a surprisingly complete picture of their stack. Wappalyzer popularised a fingerprint format for exactly this, and the open webappanalyzer project maintains those fingerprints publicly: at the time of writing, 7,613 technologies with the patterns that identify each one. The two things that make a naive implementation wrong If you write this yourself over a weekend, you will hit both of these. Technologies are injected at runtime. Take Cloudflare's own marketing site. It uses Google Tag Manager, but there is no anywhere in the HTML. Instead an inline script does this: const gtmContainerSrc = "https://www.googletagmanager.com/gtm.js?id=GTM-NDGPDFZ"; // ... var s = document.createElement('script'); s.src = gtmContainerSrc; document.head.appendChild(s); A fingerprint matcher that only looks at attributes sees nothing. The same applies to headless storefronts: Gymshark is a Shopify store, but its front end is Next.js and the only Shopify URLs on the page are in an og:image tag and a srcset attribute. The fix is to treat vendor URLs found anywhere in the document as evidence, at reduced confidence, rather than only trusting script tags. Fingerprint patterns carry metadata you have to parse correctly. Patterns look like this: (?:cdn.|.my)shopify.com;confidence:50 That trailing ;confidence:50 is a tag, not part of the regex. Split on the wrong thing and the regex keeps a trailing backslash, fails to compile, and gets silently dropped. In the database as it stands, 223 patterns carry a confidence tag and 1,392 carry a version tag, so getting this wrong costs you every version number and a good chunk of your detections. It is exactly the kind of bug that produces plausible-looking output that is quietly missing a third of the answer. Doing it without writing it I published this as an Apify Actor so you can run it on a list without maintaining any of the above: Tech Stack Detector. It costs $0.02 per website and unreachable sites are free. curl -X POST "https://api.apify.com/v2/acts/clearfetch~tech-stack-detector/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"urls": ["gymshark.com", "techcrunch.com", "notion.so"]}' Every detection comes back with the evidence behind it, so you can audit rather than trust: { "name": "Cloudflare", "confidence": 100, "categories": [{ "id": 31, "name": "CDN" }], "evidence": [ { "type": "headers", "key": "server", "pattern": "^cloudflare$", "match": "cloudflare" }, { "type": "dns", "key": "ns", "pattern": "\.cloudflare\.com", "match": "hera.ns.cloudflare.com" } ] } Filtering a thousand domains down to "everything running Shopify" is then a one-liner: from apify_client import ApifyClient client = ApifyClient("YOUR_TOKEN") run = client.actor("clearfetch/tech-stack-detector").call(run_input={"urls": domains}) for site in client.dataset(run["defaultDatasetId"]).iterate_items(): if "Shopify" in site.get("techNames", []): print(site["url"], site["company"] if "company" in site else "") Where this approach stops working Be clear about the limits before you rely on it: - No JavaScript execution. Technologies that leave no trace in the HTML, headers, cookies or DNS, and are only visible as a window global after the page boots, will be missed unless another fingerprint implies them. Running a real browser would catch those, at roughly fifty times the cost per page. - One page per site. Something that only loads on a checkout or pricing page will not show up if you only scan the homepage. Pass deeper URLs when it matters. - Confidence is not certainty. A vendor URL inside an inline script is good evidence, not proof. Raise the minimum confidence if you would rather have fewer, harder detections. Knowing those boundaries is the difference between a list you can act on and a list you have to re-check. Top comments (0)