I Built a Web Scraper That Fixes Itself When Websites Change - Then I Open-Sourced It

I spent the last month building something I couldn't find: a web scraper that doesn't break every time a website updates its HTML. Three weeks ago I open-sourced it. It's called Harvest, it's MIT-licensed, and this is the honest tour - what it does, why it exists, and the three lessons that cost me the most time.

The problem that started it

Every scraper I tried had the same failure mode: the website changes one CSS class, and suddenly you're debugging at 2 AM. Firecrawl costs $50/month. Crawl4AI (72K stars, great library) has no MCP server and no semantic cache. ScrapeGraphAI requires a paid API.

I wanted something that bypasses Cloudflare, extracts data by plain-language description (not CSS selectors), and works as an MCP server for AI agents. Free. So I built it.

Three features that changed how I scrape

1. Semantic Cache - same meaning, zero tokens

Every AI-based scraper burns tokens on repeated queries. Ask "get all prices" and then "extract product prices" - most tools process both from scratch. Harvest caches by meaning, not exact text. It uses sentence embeddings to compare queries. Same intent? Cache hit. Zero tokens.

# First call: 2K tokens consumed
harvest llm-extract https://shop.com --prompt "Get all product prices"

# Second call: instant, 0 tokens
harvest llm-extract https://shop.com --prompt "Extract prices"

# Third call: instant, 0 tokens
harvest llm-extract https://shop.com --prompt "Find prices on page"

The cache auto-invalidates when the HTML hash changes. I saw 50-70% token reduction on real workloads.

2. Self-Healing Parsers - scrapers that repair themselves

This is the one I'm proudest of. When a website changes its HTML structure, Harvest doesn't crash - it regenerates the CSS selectors via LLM.

Website updated → Old selectors fail validation → LLM gets: old HTML fragment + new HTML + old selectors → LLM returns: new working selectors → New selectors validated against schema → Selector saved. User notified.

All selector history is stored in ~/.harvest/self_healing/. You can roll back any time.

# Enable self-healing
harvest llm-extract https://shop.com --prompt "Get prices" --self-healing

3. Structural Diff - git diff for web pages

The most underrated feature. Before Harvest, when a parser broke I had no idea what changed. Now:

# Capture a snapshot
harvest snapshot https://shop.com --name v1.0

# Compare later
harvest diff https://shop.com v1.0 latest

Output:

📊 Structural Diff for https://shop.com

🆕 Added:
  • Block "Recommendations" (after description)
  • Field "Delivery" (in sidebar)

❌ Removed:
  • Field "SKU" (was in header)

🔄 Changed:
  • Price: <span class="price"> → <div class="price-container">

💡 Recommendation: Update your extractor: .price → .price-container .price-value

The Script Generator trick

This one saved me the most operational cost. One LLM call analyzes a page and generates a standalone Python script with hardcoded CSS selectors. Zero tokens at runtime. Pure Scrapling + BeautifulSoup.

# One-time: 4K tokens
harvest generate https://catalog.com --fields title price image

# Forever: 0 tokens per run
./scrape_generated.py https://catalog.com/page/1

# Batch mode
./scrape_generated.py urls.txt --csv prices.csv

Before: 1000 runs = 2M tokens ($0.30-2.00 in LLM costs). After: 1000 runs = $0.00.

Why MCP matters

Most scraping tools are libraries. Harvest runs as an MCP server - any MCP-compatible client (Claude, Cursor, Hermes, custom agents) can use it out of the box.

pip install -e ".[mcp]"
harvest-mcp

Available tools: scrape, extract, llm_extract, batch, crawl, monitor, contacts, snapshot, diff, cache-stats, generate.

Honest limitations

I'm not going to pretend this solves everything:

• Turnstile checkbox (behavioral biometrics) can still block. Regular Cloudflare JS challenges work fine.
• Requires Chromium (auto-downloaded by Scrapling, ~300MB)
• LLM extraction needs an endpoint - Ollama, OmniRoute, or any OpenAI-compatible API
• Brand new - 0 GitHub stars as of writing. No community yet.

But within those bounds, it works. I use it daily for price monitoring, content extraction, and web research.

Comparison

Quick start

pip install scrapling aiohttp
git clone https://github.com/zad111ak-ai/harvest
cd harvest
pip install -e .

# Full page content
harvest scrape https://news.ycombinator.com

# AI extraction - just describe
harvest llm-extract https://books.toscrape.com \
  --prompt "Get all book titles and prices"

# Monitor for changes
harvest monitor https://example.com/pricing

# Zero-token scraper
harvest generate https://shop.com --fields title price
./scrape_generated.py https://shop.com/page/1

What I'd do differently

If I were starting today:

• Better docs first. I wrote README after the code. Docs-first would have caught API inconsistencies earlier.
• More tests earlier. 116 tests now, but the first 30 would have prevented three regression bugs.
• Release earlier. The first public version was v0.5.0. I should have shipped at v0.1.0 and iterated.

pip install harvest-agent

Or clone from github.com/zad111ak-ai/harvest. MIT licensed. 38 commits, 9.5K lines of Python. Built in 3 weeks. Donation addresses in README if you find it useful - but the real contribution is feedback and issues.