Practical Web Scraping in Python: Playwright, Scrapy, and Where They Meet

Choosing the Right Mental Model

Before writing a line of code, ask yourself: Are you reading raw data, or are you executing a live web app?

Rule of thumb: If the target site returns clean static HTML or standard JSON endpoints, stick with Scrapy. If the text renders dynamically after page load via heavy JS framework calls, bring in Playwright.

Playwright: Clean, Explicit Automation

Playwright keeps resource management straightforward through Python context managers.

Standard Synchronous Flow

Ideal for standalone CLI tools or scripts:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com")
    print(page.title())
    browser.close()

When execution leaves the with block, Playwright cleanly shuts down the browser process-preventing orphaned Chromium instances from eating up host memory.

Asynchronous Execution

If you're integrating into an asyncio loop or web backend, use async_playwright:

import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto("https://example.com")
        print(await page.title())
        await browser.close()

asyncio.run(main())

Pro Tip: Hit APIs directly with Playwright's Network Stack

If a page exposes internal REST endpoints, don't waste CPU cycles rendering the layout engine. You can use Playwright’s underlying request context directly:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    api_context = p.request.new_context(base_url="https://jsonplaceholder.typicode.com")
    response = api_context.get("/posts/1")
    print(response.json())
    api_context.dispose()

Combining Forces: scrapy-playwright

What if you need Scrapy’s link extractors, request scheduling, item pipelines, and throttling, but your pages strictly require client-side JavaScript execution? You bridge them using scrapy-playwright.

Resolving the Event Loop Problem

Scrapy relies on Twisted, whereas Playwright runs on native asyncio. To join them, configure Scrapy to use Twisted's AsyncioSelectorReactor and set scrapy-playwright as the custom download handler.

import scrapy

class PlaywrightQuotesSpider(scrapy.Spider):
    name = "playwright_quotes"
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
    }

    def start_requests(self):
        yield scrapy.Request(
            url="https://quotes.toscrape.com/js/",
            meta={
                "playwright": True,
                "playwright_include_page": True,
            },
        )

    async def parse(self, response):
        page = response.meta["playwright_page"]
        await page.wait_for_selector("div.quote")
        content = await page.content()
        sel = scrapy.Selector(text=content)
        for quote in sel.css("div.quote"):
            print(quote.css("span.text::text").get())
        await page.close()

Architectural Gotchas & Pitfalls

• RAM Limits in Production: Running 50 concurrent Playwright browser tabs consumes gigabytes of memory compared to a few megabytes for standard Scrapy HTTP connections. Only activate meta={"playwright": True} on requests that actually require client-side JS.
• Reactor Initialization Order: The TWISTED_REACTOR setting must be configured before Twisted imports or initializes its default reactor. Setting it in custom_settings (as above) or settings.py ensures it hooks early enough.

Production Decision Matrix

• Scrapy Alone: Best for large-scale, multi-domain crawls, static HTML sites, and direct API extraction.
• Playwright Alone: Best for UI end-to-end testing, automated logins/form filling, screenshots, or single-page app extractions.
• Scrapy + Playwright: Best when you need production crawling infrastructure (queues, pipelines, proxies, throttling) but face dynamic JavaScript obstacles.

What's your go-to setup for handling heavy JavaScript rendering in Python scrapers? Do you prefer scrapy-playwright, browserless API hooks, or a custom headless setup? Drop your thoughts below!