How to set up proxies in Crawlee (JavaScript and Python)

Q: Are proxyUrls and proxy_urls the same option?

Yes. JavaScript uses camelCase ( proxyUrls , tieredProxyUrls , newUrlFunction ) and Python uses snake_case ( proxy_urls , tiered_proxy_urls , new_url_function ). The behaviour is the same.

Q: Can I use SOCKS proxies in Crawlee with Storm?

No. Storm gateways are HTTP(S) proxies only. Use http:// URLs; they carry both http and https sites.

Create a ProxyConfiguration with your gateways in proxyUrls (Python: proxy_urls), each written as http://GATEWAY_IP:PORT, and pass it to the crawler as proxyConfiguration. Then set maxConcurrency to fit your threads: one per request for HTTP crawlers, about ten per page for browser crawlers.

Updated October 2026Code & scraping frameworks

Crawlee is the open-source crawling library from Apify, with the same ideas in a Node.js version and a Python version: a request queue, automatic retries, a session pool, and crawlers that range from plain HTTP plus an HTML parser to full Playwright browsers. Proxies plug in through one object, ProxyConfiguration, which every crawler accepts.

The setup itself is short, so most questions are about behaviour. Crawlee’s docs assume a list of individual proxy IPs, and a Storm gateway is different: one IP:PORT that sends traffic out through many IPs. That changes what sessions do, what “rotation” means, and how far to push concurrency. People also run PlaywrightCrawler with the same concurrency as their Cheerio crawler and wonder why a 40-thread plan chokes.

This guide covers the setup in both languages, how sessions and proxy URLs interact with Main, 3-minute, 15-minute and residential gateways, tiered proxies, concurrency for HTTP and browser crawlers, and the errors Crawlee prints when the proxy side is wrong.

Which Storm plan fits

Crawlee’s HTTP crawlers (Cheerio, JSDOM, BeautifulSoup, Parsel) fit a rotating plan: the Main gateway gives a fresh IP per connection and 40 threads ($39/mo) covers maxConcurrency: 40. Browser crawlers need roughly ten threads per open page, so plan on 80 threads ($59) or more for 6 to 8 pages. If Crawlee’s sessions must keep one IP with their cookies, use residential ports, which hold an IP for 5 minutes.

Rotating proxies (from $14/mo): 700,000+ IPs behind fixed gateway IP:PORTs. New IP on every request, or every 3 or 15 minutes. USA, EU, USA+EU or Worldwide. Unlimited bandwidth on every plan.

Get 40 threads for $39/mo See all rotating proxies plans

Before you start

Log in to the member area and copy your gateway IP:PORTs. They never change; the rotation happens on our side.
Add the public IP of the computer or server that will run your tool under Authorized IPs, click Save, and allow up to 15 minutes before testing. Rotating and residential proxies use IP authentication, so there is no username or password.
Dedicated proxies work with either IP authentication or a username and password. Use user:pass if your IP changes or the tool runs on several machines.
Count your threads: the tool’s total open connections must stay within your plan (for example 40 threads on the 40-thread plan).

On this page

Set up proxies in Crawlee, step by step
Copy-paste Crawlee examples
Sessions vs rotation when the proxy is a gateway
Concurrency math: HTTP crawlers vs browser crawlers
Which Storm plan for a Crawlee project
Running Crawlee on servers, Docker and cloud platforms
Common errors and fixes
FAQ

Set up proxies in Crawlee, step by step

Authorize the IP that runs the crawler
Rotating and residential gateways accept only authorized IPs. Add the public IP of your laptop or server under Authorized IPs in the member area, save, and allow up to 15 minutes. A crawler deployed to a platform with changing outbound IPs won’t pass this check; see the last section for options.
Pick gateways for the job
Main gateway for a new IP on every connection, 3-minute or 15-minute gateway when a session must keep one IP, residential ports for residential IPs that change every 5 minutes. Copy each IP:PORT from the member area.
Create the ProxyConfiguration
JavaScript: new ProxyConfiguration({ proxyUrls: [...] }) from crawlee. Python: ProxyConfiguration(proxy_urls=[...]) from crawlee.proxy_configuration. Write every URL with http://, even for https sites.
Pass it to the crawler
Add proxyConfiguration (Python: proxy_configuration=) to the crawler options. It works the same for CheerioCrawler, HttpCrawler, PlaywrightCrawler, BeautifulSoupCrawler and the rest.
Set concurrency from your thread count
JavaScript: maxConcurrency on the crawler. Python: concurrency_settings=ConcurrencySettings(max_concurrency=...). HTTP crawlers: up to your threads. Browser crawlers: threads divided by ten.
Log the proxy in the handler
Read proxyInfo in the JS requestHandler, or context.proxy_info in Python. Print proxyInfo.url with the status code so you can see which gateway or port each block came from.
Run a small crawl and check the IPs
Queue a few requests to https://api.ipify.org?format=json with unique keys and print the response. Pool IPs mean the proxy works. Then raise concurrency step by step while the error rate stays flat.

Copy-paste Crawlee examples

Replace GATEWAY_IP:PORT with gateways from your member area. No username or password is needed on rotating or residential gateways once your IP is authorized.

JavaScript: CheerioCrawler on the Main gateway, 40 in flight

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: ['http://GATEWAY_IP:PORT'],   // Main gateway: new IP per connection
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    maxConcurrency: 40,              // = threads on the 40-thread plan
    maxRequestRetries: 3,
    useSessionPool: true,
    persistCookiesPerSession: true,
    async requestHandler({ request, body, proxyInfo }) {
        console.log(proxyInfo?.url, request.url, body.length);
    },
});

await crawler.run(['https://crawlee.dev/']);

JavaScript: PlaywrightCrawler, 4 pages on 40 threads

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';

const crawler = new PlaywrightCrawler({
    proxyConfiguration: new ProxyConfiguration({
        proxyUrls: ['http://GATEWAY_IP:PORT'],   // 15-minute gateway for logged-in flows
    }),
    maxConcurrency: 4,                 // ~10 threads per open page
    navigationTimeoutSecs: 60,
    async requestHandler({ page, request, proxyInfo }) {
        console.log(proxyInfo?.url, request.url, await page.title());
    },
});

await crawler.run(['https://crawlee.dev/']);

JavaScript: sessions pinned to residential ports

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

// 5 residential ports = 5 exit IPs at once, each changes every 5 minutes
const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: [1, 2, 3, 4, 5].map((n) => `http://GATEWAY_IP:PORT${n}`),
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    maxConcurrency: 50,                // 10 per port, well under 50 per port
    useSessionPool: true,
    persistCookiesPerSession: true,
    sessionPoolOptions: {
        maxPoolSize: 25,
        sessionOptions: { maxAgeSecs: 300 },   // retire cookies when the port IP turns over
    },
    async requestHandler({ request, session, proxyInfo }) {
        console.log(session?.id, proxyInfo?.url, request.url);
    },
});

Python: BeautifulSoupCrawler with ConcurrencySettings

import asyncio

from crawlee import ConcurrencySettings
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration


async def main() -> None:
    proxy_configuration = ProxyConfiguration(
        proxy_urls=['http://GATEWAY_IP:PORT'],   # http:// even for https sites
    )
    crawler = BeautifulSoupCrawler(
        proxy_configuration=proxy_configuration,
        concurrency_settings=ConcurrencySettings(max_concurrency=40),
        max_request_retries=3,
    )

    @crawler.router.default_handler
    async def handler(context: BeautifulSoupCrawlingContext) -> None:
        context.log.info(f'{context.proxy_info.url} {context.request.url}')

    await crawler.run(['https://crawlee.dev/'])


asyncio.run(main())

Tiered: rotating first, residential when a site blocks

// JavaScript. Crawlee starts on tier 0 and moves up when it detects blocking.
const proxyConfiguration = new ProxyConfiguration({
    tieredProxyUrls: [
        ['http://GATEWAY_IP:PORT'],                                  // rotating Main gateway
        ['http://GATEWAY_IP:PORT1', 'http://GATEWAY_IP:PORT2'],      // residential ports
    ],
});
// Python: ProxyConfiguration(tiered_proxy_urls=[[...], [...]])

Sessions vs rotation when the proxy is a gateway

Crawlee’s session pool ties each session (cookies plus some headers) to one proxy URL: newUrl(sessionId) returns the same URL for the same session. Without a session it hands out proxyUrls round-robin. That model assumes one URL equals one IP. With Storm it depends on the gateway:

Main rotating gateway: one URL, but a new IP per connection. Every session gets the same URL, and each session’s cookies may leave from many IPs. That’s fine for stateless scraping. Turn persistCookiesPerSession off if the site punishes cookies that jump between IPs.
3-minute and 15-minute gateways: the IP holds for that window, so a session’s requests stay on one IP for a while. Set sessionOptions.maxAgeSecs near the window (180 or 900) so Crawlee drops cookies around the time the IP changes.
Residential ports: list each port as its own URL. Sessions spread over the ports, and every port is one IP for 5 minutes, so maxAgeSecs: 300 keeps sessions and IPs roughly in step.

By default the session pool treats 401, 403 and 429 as blocked and retires the session. With a single Main gateway URL the next session still uses the same URL, but it gets a new connection and therefore a new IP, so retiring still helps. If every retry is blocked, the target is rejecting datacenter IPs in general; see the tiered setup above or rotating vs static proxies.

Concurrency math: HTTP crawlers vs browser crawlers

Crawlee autoscales between minConcurrency and maxConcurrency based on CPU and memory. It knows nothing about your proxy plan, and the JS default ceiling is 200. Always set the ceiling yourself.

Cheerio, JSDOM, Http, BeautifulSoup, Parsel: one request is about one connection. 40 threads means maxConcurrency: 40; 150 threads means up to 150.
Playwright and Puppeteer crawlers: a page loads HTML, scripts, images, fonts and trackers over many connections at once, around 10 threads per page. 40 threads runs about 4 pages, 80 threads about 8, 200 threads about 20.
Google and other search engines: Main gateway and no more than a quarter of the plan. On 40 threads, that’s maxConcurrency: 10 with an HTTP crawler, or 1 page with a browser crawler.

Two ways to stretch a browser plan: block heavy resources in a preNavigationHooks function with Crawlee’s blockRequests helper, or render only the pages that need JavaScript and fetch the rest with Cheerio. maxRequestsPerMinute (Python: max_tasks_per_minute) caps the pace for sites that rate-limit, separately from concurrency. More on counting in the threads guide.

Which Storm plan for a Crawlee project

HTML pages, listings, product data: rotating proxies on the Main gateway. 40 threads for $39/mo, 80 for $59, 150 for $97. Bandwidth is unlimited, which matters when a browser crawler pulls every script and image.
Browser crawls of heavy sites: size by pages: 8 pages need about 80 threads, 15 pages about 150.
Sites that block datacenter IPs, or flows that need cookies and IP to match: residential ports. 5 ports ($50/mo) give 5 IPs at once with up to 50 threads each. Note there’s no city or single-country targeting: USA or EU.
Trying it first: the 10-thread package ($14/mo) runs a Cheerio crawler at 10 or one browser page. The smallest package of each proxy type has a 24-hour money-back guarantee on your first order.

Running Crawlee on servers, Docker and cloud platforms

The gateway sees the public IP the crawler’s traffic leaves from. On a VPS or in Docker on that VPS, authorize the server’s IP once. Platforms that run each job on a different machine with a different outbound IP (serverless functions, many hosted CI runners, managed actor platforms) can’t be authorized by IP, so rotating and residential gateways will reset those connections.

The fixes: run the crawler on a server with a fixed IP, or use dedicated proxies with credentials in the URL (http://USERNAME:PASSWORD@PROXY_IP:PORT), which Crawlee passes through as-is. The IP authentication guide shows how to authorize a dynamic DNS hostname for a home connection.

Common errors and fixes

ECONNRESET, socket hang up or Python Server disconnected on every requestYour IP isn’t authorized, or it was saved less than 15 minutes ago. Storm rotating and residential gateways reset unauthorized connections instead of sending a 407. Check the crawler machine’s public IP against Authorized IPs.

net::ERR_PROXY_CONNECTION_FAILED or ERR_TUNNEL_CONNECTION_FAILED in a browser crawlerThe browser couldn’t use the proxy: wrong IP or port, an unauthorized IP, or an https:// proxy URL. See the tunnel error guide.

ECONNREFUSED GATEWAY_IP:PORTNothing is listening on that address. Recopy the gateway and check outbound firewall rules.

Request blocked - received 403 status codeThe proxy works and the site refused the request. Lower concurrency, add maxRequestsPerMinute, keep sessions on, and move strict sites to a residential tier.

Timeouts climb as the crawl speeds upAutoscaling pushed past your thread count, or a browser crawler runs too many pages. Set maxConcurrency from the math above.

407 Proxy Authentication RequiredDedicated proxy credentials are wrong or have unescaped characters (encode @ as %40). On rotating or residential gateways a 407 comes from another proxy, often an HTTP_PROXY variable.

Invalid URL error, or the proxy seems ignoredEach entry must be a full URL with a scheme: http://GATEWAY_IP:PORT, not GATEWAY_IP:PORT.

FAQ

Do I need username and password in Crawlee proxyUrls for Storm rotating proxies?

No. Rotating and residential gateways authorize your IP, so each entry is http://GATEWAY_IP:PORT. Only dedicated proxies use http://USERNAME:PASSWORD@PROXY_IP:PORT, and they also accept IP authorization.

Does Crawlee rotate the IP on every request with one gateway URL?

Crawlee itself only rotates between the URLs you list. The IP rotation happens on the gateway: on the Main gateway every new connection gets a new IP, while the 3-minute, 15-minute and residential gateways keep the IP for their window.

What maxConcurrency should I use with PlaywrightCrawler?

Your threads divided by about ten: 4 on the 40-thread plan, 8 on 80 threads, 15 on 150. Pages full of ads and video need more; pages with heavy resources blocked need fewer.

Are proxyUrls and proxy_urls the same option?

Yes. JavaScript uses camelCase (proxyUrls, tieredProxyUrls, newUrlFunction) and Python uses snake_case (proxy_urls, tiered_proxy_urls, new_url_function). The behaviour is the same.

Can I use SOCKS proxies in Crawlee with Storm?

No. Storm gateways are HTTP(S) proxies only. Use http:// URLs; they carry both http and https sites.

Which Storm plan should I start with for Crawlee?

For Cheerio or BeautifulSoup crawls, 40 rotating threads. For Playwright crawls, 80 threads or more. For sites that block datacenter IPs, 5 residential ports. You can upgrade or move to a smaller plan in the member area in one click, and the unused part of the current plan is credited.

Still have questions? Contact us here. A real person answers.

Tool facts checked against the official documentation (October 2026): Crawlee JS: Proxy management · Crawlee JS: Scaling crawlers · Crawlee JS: Session management · Crawlee JS: SessionPoolOptions · Crawlee JS: SessionOptions · Crawlee Python: Proxy management · Crawlee Python: Scaling crawlers. Storm Proxies facts: our plans page and refund policy.

Unlimited bandwidth. One flat monthly price.

Access is live the moment you pay, and the smallest package of each proxy type has a 24-hour money-back guarantee on your first order.

Get 40 threads for $39/mo Compare all plans