Stop Spinning Up Headless Chrome for Public JSON: HTTP-First Extraction Patterns for Shopify, Multi-ATS Boards, and Substack (with n8n)
A common source of fragility is a cron job that starts a headless browser to parse HTML cards on a site that already serves structured public JSON.
When a store renames a CSS class or loads prices asynchronously, DOM selectors break silently, browser memory spikes inside small n8n containers, and empty arrays look identical to HTTP 429 rate limits.
These three HTTP-first patterns cover catalog changes, incomplete runs, and feeds with published transcripts.
Shopify Storefronts: Diff /products.json by Variant ID, Never by Page Card
Some Shopify storefronts expose a paginated JSON feed at the catalog level. Check availability and usage terms for the store you plan to monitor:
GET https://<store-domain>/products.json?limit=250&page=1
Three rules prevent silent monitor bugs when polling /products.json:
- Key your state on immutable variant IDs, not titles or handles. When a single size (M or EU 42) restocks or changes price, the parent product title and URL never move - only the nested
variants[].id,variants[].available, andvariants[].price/compare_at_priceflip. - Treat http:// pasted URLs, percent-encoded handles, and Shopify Markets locales carefully. Normalizing pasted store URLs to
https://avoids mixed-content blocks, decoding collection handles preserves non-ASCII paths cleanly, and keeping Markets locale prefixes (like/fr/or/en-ca/) stops price checks from reading the wrong currency feed. Virtual collection paths (/collections/vendors?...,/collections/all) also behave differently from handle feeds (/collections/{handle}/products.json). - Never confuse an HTTP 429 Too Many Requests with an empty catalog. Honor
Retry-Afterheaders for a bounded backoff, and flag throttled runs as incomplete (rate_limited) instead of overwriting last-known stock with zero items.
Multi-ATS Job Boards: Separate "Zero Openings" from Lookup Failures
Many public job boards on Greenhouse, Lever, Ashby, Workable, or SmartRecruiters expose structured job listings. Check which endpoint is available for each provider before choosing a browser workflow.
One failure mode in job trackers is overwriting state on a partial failure:
- If source A returns 220 jobs cleanly and source B times out or returns HTTP 403, overwriting your snapshot as 0 open roles for source B triggers bogus "closed" removals on the current run and bogus "new role" alerts on the next healthy run.
- Always pair
(ats_provider, board_slug, immutable_job_id)as the deduplication key (titles and URL slugs get edited mid-posting) and isolate per-board error states from genuine empty boards (200 OK with[]).
Substack & Audio Archives: Exhaust XML/JSON Metadata Before Touching Whisper or Browsers
Two checks can avoid a browser launch or an audio transcription job when the publisher already provides the content:
- Substack: RSS feeds are great for new issues, while archive endpoints/sitemaps let you backfill older posts as clean text/Markdown without rendering full browser pages (preserving inline links and separating subscriber-only previews from full free essays cleanly).
- Podcast transcripts: Always inspect
<podcast:transcript>tags inside the show's RSS<item>nodes before scheduling an audio download + Whisper pass. Shows that use the tag can link to timestamped VTT/SRT or JSON transcripts directly from the feed.
Plug-and-Play n8n Templates & Hosted Pay-Per-Event Endpoints
We open-sourced MIT-licensed n8n JSON workflow templates (Shopify Price Monitor, Multi-ATS Startup Jobs Tracker, Vinted Underpriced Comps Alert) plus Claude Desktop / Cursor MCP configs on GitHub:
- Open Templates & MCP Configs: https://github.com/atelierprojetscontact/pulse-extract-templates
- Hosted HTTP-first Actors & Example Tasks (pay-per-event; check each listing for current pricing):
- Multi-ATS Jobs Scraper
- Shopify Catalog, Price & Stock Monitor
- Vinted EU & UK Catalog & Price Comps Scraper
- Substack Newsletter & Archive Scraper
- Podcast & Audio RSS Transcript Extractor
Example cURL request to a hosted endpoint that returns normalized JSON. Running the request uses your Apify account and can incur charges:
curl -X POST \
"https://api.apify.com/v2/acts/pulse_extract~multi-ats-jobs-scraper/run-sync-get-dataset-items" \
-H "Authorization: Bearer $APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"targets":["ashby:linear"],"maxJobsPerCompany":5,"includeDescription":false}'
Keep heavy HTML or JSON diffing out of your main n8n memory process, persist explicit failure states separate from empty arrays, and reserve browser rendering for pages that need client-side JavaScript and permit automated access.
Comments
No comments yet. Start the discussion.