How to set up a proxy in Scrapy

Scrapy is the Python framework people pick when a script with requests stops being enough: it schedules thousands of URLs, runs them concurrently on an event loop, follows links, retries failures and exports items. Proxy support is already inside it. A downloader middleware called HttpProxyMiddleware is enabled by default and routes any request that carries a proxy key in its meta.

The setup is short. The trouble usually starts later: the spider runs 16 requests at once by default (or far more after someone raises the setting), a 10-thread test plan can’t keep up, retries pile onto a site that’s already rate-limiting, and a spider that should rotate keeps one IP because Scrapy holds connections open. Each of those has a clean fix in settings.

Below you’ll find four ways to attach a proxy, a reusable middleware for one or many gateways, the settings that match Scrapy’s concurrency to your plan, how retries and AutoThrottle behave behind a rotating gateway, and how to keep the same setup when you add scrapy-playwright for JavaScript pages.

Storm plans limit how many connections are open at the same moment. Scrapy’s matching knob is CONCURRENT_REQUESTS: the most requests the downloader handles at once across every domain. Its built-in default is 16. That already overloads the 10-thread test package and leaves most of a 150-thread plan unused.

• 10 threads (test package): CONCURRENT_REQUESTS = 8. Enough to prove the spider works, not to run it in production.
• 40 threads: 36–40. 80 threads: 72–80. 150 threads: 135–150. 200 threads: 180–200.
• Two spiders on one plan: split the number. Two spiders at 40 each on a 40-thread plan means 80 connections against a 40 limit, and roughly half of them fail.
• Search engine result pages: no more than 25% of the plan, through the Main gateway. A 40-thread plan gets CONCURRENT_REQUESTS = 10 for that spider.

Then set CONCURRENT_REQUESTS_PER_DOMAIN for politeness. Projects created with a recent scrapy startproject ship with a per-domain value of 1 and a 1-second delay, which is why a fresh single-site spider can feel slow even on a big plan. Raise those two deliberately, per site, rather than only cranking the global number.

The Main gateway assigns a new exit IP when a connection or HTTPS tunnel is opened. Scrapy’s HTTP/1.1 handler keeps a pool of persistent connections and reuses them, so a burst of requests to one site may travel through the same tunnel and share an IP. Usually that’s harmless; the pool spreads load over many connections anyway.

If a test with five quick requests to httpbin shows the same IP twice, that’s connection reuse, not a stuck gateway. Spread the same test over several domains or more time and the IPs vary. When you need the opposite, the same IP across a login and the pages behind it, don’t fight the Main gateway. Route that spider through the 3- or 15-minute gateway, or a residential port, where the IP holds by design. Note that the 3-minute gateway is meant for sign-ups, social sites and browsing, not for search engine scraping.

RetryMiddleware is on by default and retries twice for codes 500, 502, 503, 504, 522, 524, 408 and 429, plus timeouts and dropped connections. Behind a rotating gateway a retry is useful: whenever it goes out on a new connection, it leaves from a different IP. The default RETRY_PRIORITY_ADJUST of -1 also schedules a retry behind fresh requests, which gives a rate-limited site a short break before it sees the URL again. Three retries is a sensible ceiling. Beyond that you’re usually hammering a page that’s gone or a site that wants you to slow down.

AutoThrottle adjusts the delay from measured latency. It starts at AUTOTHROTTLE_START_DELAY (5 seconds by default), never goes below DOWNLOAD_DELAY, and aims for AUTOTHROTTLE_TARGET_CONCURRENCY parallel requests per site (1.0 by default). Error responses can raise the delay but never lower it, which is exactly what you want when a site starts returning 503s. Two things to remember: AutoThrottle works per site, not across your plan, so CONCURRENT_REQUESTS still has to respect your thread limit; and the per-domain cap still applies, so a target concurrency above it is never reached.

A gateway IP:PORT never changes, and the rotation happens on our side. So in most projects you don’t need a proxy list, a rotation package or a “ban detection” plugin at all. One Main gateway already sends traffic out through the whole pool.

Packages like scrapy-rotating-proxies exist for lists of static proxies, where the code itself must take a dead proxy out of rotation. You only need that pattern with dedicated proxies: give the middleware above your list of dedicated IP:PORTs and it spreads requests over them. For residential, list one port per entry; each port is a separate IP, so 10 ports in the list means requests leave from 10 IPs at the same moment.

Mixing gateways inside one project is common: catalog spiders on the Main gateway, an account spider on the 15-minute one. Set custom_settings or a class attribute per spider, and have the middleware read it instead of the global list.

When a page builds its content in the browser, add scrapy-playwright. Its download handler ignores meta['proxy'], which is a documented limitation. Set the proxy in PLAYWRIGHT_LAUNCH_OPTIONS for the whole browser, or define named contexts in PLAYWRIGHT_CONTEXTS, each with its own proxy, and pick one per request with meta={'playwright': True, 'playwright_context': 'eu'}.

Budget threads differently here. A rendered page loads scripts, images and API calls in parallel, around 10 connections per open page. Cap pages with PLAYWRIGHT_MAX_PAGES_PER_CONTEXT and PLAYWRIGHT_MAX_CONTEXTS so that pages × 10 stays inside your plan. On a 40-thread plan that’s about 4 pages, or fewer if plain Scrapy requests share the plan. The Playwright guide covers browser-level setup in more depth.