Stop Rewriting Your Web Scraper for Every New Site

Every web scraper I've ever written started the same way: a quick script for one site, hardcoded selectors, done in an afternoon.

Then someone asks for "the same thing, but for this other site" - and you copy the file, change the URL, re-hunt every selector, rediscover the pagination quirks, and ship a second script that shares 90% of its DNA with the first. By the fourth site you're maintaining a small zoo of near-identical programs, each of which breaks independently whenever a site redesigns.

This post builds the honest version of that first script - small, readable, actually working - and then shows where it hits its ceiling, and the config-driven pattern I now use instead.

The demo: one file, one site, twenty books

The target is books.toscrape.com, a practice site built for exactly this. Goal: every book on the first catalogue page - title, price, rating, availability - into a CSV.

The whole thing is one file, scrape.py, using only requests and BeautifulSoup:

URL = "https://books.toscrape.com/catalogue/page-1.html"
HEADERS = {"User-Agent": "Mozilla/5.0 ... web-scraper-demo/1.0"}
response = requests.get(URL, headers=HEADERS, timeout=30)
response.raise_for_status()
response.encoding = "utf-8" # don't let requests guess - see below

Two details in those few lines matter more than the rest of the script combined. First, send a real User-Agent and keep your request rate polite; scraping that hammers a server is how IPs get banned and side projects get legal letters.

Second, the encoding line: on my first run, every price came out as £51.77 instead of £51.77, because requests trusted the response headers over the page's actual UTF-8 content. One explicit line fixed all twenty rows. Small bug, but it's the kind that silently poisons a dataset if you're not eyeballing your output.

Parsing is one function. Each book is an article.product_pod card, and everything we want hangs off it:

for pod in soup.select("article.product_pod"):
    title = pod.select_one("h3 a")["title"]
    price = pod.select_one("p.price_color").get_text(strip=True)
    rating_el = pod.select_one("p.star-rating")
    rating = next(c for c in rating_el["class"] if c in RATING_WORDS)
    availability = " ".join(
        pod.select_one("p.availability").get_text(" ", strip=True).split()
    )

The rating is a nice example of real-world markup: it isn't text at all, it's a CSS class (class="star-rating Three"), so you extract it from the class list rather than the element's contents. The availability text comes with stray whitespace and a nested icon element, so it gets normalized with a split-and-join.

Then csv.DictWriter writes the rows out, and you have books.csv with 20 clean records. Total: about 60 lines. It works. Ship it?

Where the one-file scraper breaks

It breaks at the second site, and the failures are structural, not incidental:

• Selectors live in code. A new site means editing Python, re-testing, and redeploying - for what is really just data about the site.
• Pagination is a snowflake. This demo deliberately stops at page 1. Real sites paginate with next-links, page-number URLs, or infinite scroll, and each style needs different driving logic.
• Not everything is static HTML. The moment a site renders with JavaScript, requests gets an empty shell and your selectors match nothing - silently.
• Nobody counts the bad rows. Real pages have missing prices, duplicate listings, layout variants. A quick script mixes them into your CSV and you find out downstream, in a spreadsheet, at the worst time.

None of these are hard problems individually. Together they mean the "quick script" quietly becomes the job.

The fix: the site is data, not code

The pattern I use in production flips the script inside out. Everything site-specific - start URLs, the item container, per-field selectors and types, pagination style, rate limits, dedup keys - moves into a YAML config. The engine stays fixed. A new site looks like this:

site:
  name: books
  start_urls: ["https://books.toscrape.com/catalogue/page-1.html"]
  rate_limit_seconds: 1.0
items:
  container: "article.product_pod"
  fields:
    title: { css: "h3 a", attr: "title", type: string, required: true }
    price: { css: "p.price_color", type: float, required: true }
pagination:
  type: next_link
  next_selector: "li.next a"
dedup:
  key_fields: [title]

Same target, same fields - but now adding a site is writing twenty lines of YAML instead of forking a program. The engine adds the things the demo lacks: all three pagination styles, a Playwright mode for JS-rendered pages, type coercion and required-field validation, deduplication with a count of dropped rows (so bad data is a number in your log, not a surprise in your spreadsheet), Excel output, and retries with backoff so it can run unattended from cron.

Try it, then skip the rewrite

The demo from this post is free and MIT-licensed - clone it, run it, take it apart:

👉 GitHub: https://github.com/mahuan2928/web-scraper-demo

And if you're at the "second site" stage - or you can see it coming - the full config-driven engine is available here, with example configs (including this same book site, all 50 pages of it), the Playwright mode, Sheets export, and the validation/dedup pipeline described above:

👉 Config-Driven Web Scraper Engine ($39): https://mahuan220.gumroad.com/l/zknkka

Build the demo version once to learn the mechanics. Just don't build it five times.

• Marty Ma, Senior AI & Data Engineer (Tokyo)