Scrapy vs Selenium: Choosing a Python Web Scraping Tool

Web · Published 2026-10-24

A spider's web covered in dew between dry stems at dawn — A dew-covered spider's web strung between dry stems.

Use Scrapy when you need to crawl many pages efficiently and the data is in the HTML (or in JSON endpoints the page calls). Use Selenium (or, increasingly, Playwright) when the content only exists after JavaScript runs in a real browser or you must interact with the page by clicking, typing or logging in. For large crawls of JavaScript-heavy sites, combine them: Scrapy for scheduling and pipelines, a headless browser only for the pages that need it.

Two different kinds of tool

The comparison is a little unfair, because the two projects solve different problems.

Scrapy is a crawling framework. It sends HTTP requests directly, parses the returned HTML with CSS or XPath selectors, and manages everything around that: an asynchronous scheduler (built on Twisted), request deduplication, retries, redirects, cookies, throttling, robots.txt handling, and export to JSON, CSV or databases. It never executes the page's JavaScript.

Selenium is a browser automation library. Through the W3C WebDriver protocol it controls a real Chrome, Firefox, Edge or Safari: it loads pages, runs their JavaScript, clicks and types like a user, and lets you read the resulting DOM. It was built for testing web applications; scraping is a side use. It has no crawler, scheduler or data pipeline. Those are up to you.

Comparison at a glance

Aspect	Scrapy	Selenium
Core model	Asynchronous HTTP crawler framework	Remote control of a real browser
JavaScript	Not executed (add scrapy-playwright if needed)	Fully executed
Throughput	Tens to hundreds of pages per second per process, server permitting	Roughly one page every one or more seconds per browser
Memory	Tens of MB for the whole crawler	Hundreds of MB per browser instance
Concurrency	Built in (`CONCURRENT_REQUESTS`)	Manual: multiple drivers, threads or Selenium Grid
Crawling, dedup, retries	Built in	Write it yourself
Data export	Feed exports and item pipelines	Write it yourself
User interaction	Form posts only (`FormRequest`)	Clicks, typing, scrolling, file uploads, screenshots
Typical use	Catalogs, news archives, large-scale structured crawls	Login-heavy portals, single-page apps, small scrapes, end-to-end testing

The throughput figures are orders of magnitude, not benchmarks: real speed depends on the target server, your politeness settings and page weight.

JavaScript rendering: the deciding question

Before choosing, open the page with JavaScript disabled, or run curl on the URL and search the output for the data you want. Three outcomes are possible:

The data is in the HTML. Scrapy is the right tool.
The data arrives via an XHR/fetch call (visible in the browser's developer tools, Network tab). Request that JSON endpoint directly from Scrapy. This is usually faster and more stable than rendering anything.
The data is computed in the browser or guarded by interactive flows. You need a browser: Selenium, Playwright, or Scrapy with a browser download handler.

Many developers reach for Selenium by default and end up paying the cost of a browser for pages that a plain HTTP request could have fetched.

A minimal Scrapy spider

This spider extracts products from a paginated listing and follows the "next" link. Politeness settings are declared on the spider itself. It was run with Scrapy 2.19 against a small local test site:

import scrapy


class ProductsSpider(scrapy.Spider):
    name = "products"
    start_urls = ["http://127.0.0.1:8765/page1.html"]
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "AUTOTHROTTLE_ENABLED": True,
        "DOWNLOAD_DELAY": 0.5,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 4,
        "USER_AGENT": "example-research-bot/1.0 (+https://example.com/bot)",
    }

    def parse(self, response):
        for product in response.css("article.product"):
            yield {
                "name": product.css("h2::text").get(),
                "price": float(product.css("span.price::text").get()),
                "url": response.url,
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

scrapy runspider products_spider.py -O products.json

[
{"name": "Item 1-0", "price": 10.99, "url": "http://127.0.0.1:8765/page1.html"},
{"name": "Item 1-1", "price": 11.99, "url": "http://127.0.0.1:8765/page1.html"},
{"name": "Item 1-2", "price": 12.99, "url": "http://127.0.0.1:8765/page1.html"},
{"name": "Item 2-0", "price": 20.99, "url": "http://127.0.0.1:8765/page2.html"},
{"name": "Item 2-1", "price": 21.99, "url": "http://127.0.0.1:8765/page2.html"},
{"name": "Item 2-2", "price": 22.99, "url": "http://127.0.0.1:8765/page2.html"}
]

-O overwrites the output file; -o appends. Real projects usually start from scrapy startproject.

Middlewares and pipelines

Scrapy's extension points are what make it a framework rather than a library. Downloader middlewares sit between the engine and the network: they can rotate proxies or headers, retry on specific status codes, or hand certain requests to a browser. Item pipelines receive every scraped item for validation, cleaning, deduplication and storage:

from scrapy.exceptions import DropItem


class CleanPricePipeline:
    # Recent Scrapy versions deprecate the required `spider` argument;
    # making it optional works on both old and new releases.
    def process_item(self, item, spider=None):
        if item.get("price") is None:
            raise DropItem(f"missing price: {item!r}")
        item["price"] = round(item["price"], 2)
        return item

Enable it with ITEM_PIPELINES = {"myproject.pipelines.CleanPricePipeline": 300}; the number sets the order. With Selenium, you write the equivalent plumbing yourself.

A minimal Selenium scraper

The same task with Selenium 4. Since Selenium 4.6, the bundled Selenium Manager locates or downloads a matching driver automatically, so you no longer have to install chromedriver by hand:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--window-size=1280,900")

driver = webdriver.Chrome(options=options)
try:
    driver.get("http://127.0.0.1:8765/page1.html")
    wait = WebDriverWait(driver, timeout=10)
    while True:
        wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "article.product")))
        for product in driver.find_elements(By.CSS_SELECTOR, "article.product"):
            name = product.find_element(By.CSS_SELECTOR, "h2").text
            price = float(product.find_element(By.CSS_SELECTOR, "span.price").text)
            print(name, price)
        next_links = driver.find_elements(By.CSS_SELECTOR, "a.next")
        if not next_links:
            break
        next_links[0].click()
        wait.until(EC.staleness_of(next_links[0]))   # wait for the old page to go away
finally:
    driver.quit()

Item 1-0 10.99
Item 1-1 11.99
Item 1-2 12.99
Item 2-0 20.99
Item 2-1 21.99
Item 2-2 22.99

Tested with Selenium 4.51 and headless Chrome. When running as root inside a Docker container, Chrome usually also needs --no-sandbox (and often --disable-dev-shm-usage) to start; avoid those flags on a normal desktop. Note the explicit waits: in a browser, content appears asynchronously, and reading the DOM too early is the main cause of flaky Selenium scrapers. Prefer WebDriverWait with expected conditions over time.sleep(), and always call driver.quit() or you will leak browser processes.

Speed, resources and scaling

A Scrapy process keeps many requests in flight on a single thread and only downloads HTML. A browser downloads and executes everything a page references (scripts, stylesheets, fonts, images, trackers), then builds a DOM and lays it out. That difference dominates at scale: a crawl of 100,000 pages is routine for one Scrapy process on a small server, while the same job with browsers needs a fleet of instances, Selenium Grid or a cloud browser service.

Scrapy's scaling knobs are settings: CONCURRENT_REQUESTS, CONCURRENT_REQUESTS_PER_DOMAIN, AUTOTHROTTLE_* and JOBDIR for pausing and resuming long crawls. Selenium scales by running more browsers, which means more CPU and memory linearly.

Combining them: scrapy-playwright and friends

You don't have to pick one for a whole project. scrapy-playwright is a download handler that renders selected requests in a Playwright-controlled browser while keeping Scrapy's scheduler, middlewares and pipelines:

import scrapy


class JsProductsSpider(scrapy.Spider):
    name = "js_products"
    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
    }

    async def start(self):          # Scrapy 2.13+; older versions use start_requests()
        yield scrapy.Request("https://example.com/catalog", meta={"playwright": True})

    def parse(self, response):
        # response.text is the DOM after the browser ran the page's JavaScript
        for product in response.css("article.product"):
            yield {"name": product.css("h2::text").get()}

Install it with pip install scrapy-playwright followed by playwright install chromium. Only requests with meta={"playwright": True} use the browser; everything else stays plain HTTP, so you pay the rendering cost only where it is necessary. A version note: Scrapy 2.13 introduced the asynchronous start() method; in our test with Scrapy 2.19, a spider that defined only start_requests() sent no requests at all, so define start() (plus start_requests() if you must support Scrapy older than 2.13).

Playwright as the modern alternative to Selenium

For new browser-based scrapers, Playwright for Python is worth considering over Selenium. It auto-waits for elements before acting, supports Chromium, Firefox and WebKit from one API, offers both sync and async interfaces, and can intercept network traffic, which lets you capture the JSON responses a page loads instead of parsing the DOM. Selenium remains the better choice when you need its broad browser and vendor support, an existing Selenium Grid, or compatibility with an established test suite.

Ethics, robots.txt and rate limiting

Whichever tool you use, scrape responsibly:

Respect robots.txt. Scrapy projects generated with startproject set ROBOTSTXT_OBEY = True; keep it. Selenium does not check robots.txt, so use urllib.robotparser yourself.
Rate-limit. Use DOWNLOAD_DELAY and AutoThrottle in Scrapy; add delays between page loads in Selenium. Back off on HTTP 429 and 503.
Identify yourself with an honest User-Agent that includes contact information.
Read the terms of service, prefer official APIs when they exist, and be careful with personal data, which is regulated in many jurisdictions (GDPR in the EU, for example).

Common pitfalls

Empty selectors in Scrapy: the content is rendered by JavaScript. Look for the underlying API call before adding a browser.
Flaky Selenium runs: replace sleeps with explicit waits; handle StaleElementReferenceException by re-locating elements after navigation.
Driver/browser mismatch: an old chromedriver on your PATH can take precedence over Selenium Manager and fail with "session not created". Remove it and let Selenium Manager resolve the driver.
Blocking calls inside Scrapy callbacks: calling requests or Selenium synchronously inside a spider freezes the whole asynchronous engine.

Summary

Scrapy is a fast, resource-light crawling framework with scheduling, retries, throttling and pipelines built in, but it does not run JavaScript. Selenium drives a real browser, so it handles dynamic pages and user interactions at a much higher cost per page and without any crawling infrastructure. Check first whether the data is reachable through plain HTTP or a JSON endpoint; if it is, use Scrapy. If it truly needs a browser, use Selenium or Playwright for small jobs and scrapy-playwright when you need both rendering and scale.