Which proxies are best for web scraping?

You’re here because your scraper works on your laptop for a hundred pages, then starts getting 403s, CAPTCHAs or empty responses once you scale it. Or because you’re about to buy proxies and every provider says theirs is “the best for scraping”. The honest answer is that no proxy type is best everywhere. The right choice depends on three things: how strictly the target site treats datacenter IPs, whether your scraper needs to stay logged in, and how many pages you need per hour.

The usual mistakes are expensive in different ways. Buying residential bandwidth for sites that would happily serve datacenter IPs costs you money on every gigabyte. Running a big crawl through a handful of static IPs gets those addresses rate-limited in an afternoon. And running more parallel connections than your plan allows looks exactly like “the proxies are not working”, which is the most common complaint we hear.

This guide gives you a decision path, the cost math behind per-GB versus per-thread pricing, worked sizing for real job sizes, a test script to compare proxy types on your own targets, and the errors that tell you which way to adjust.

• Rotating datacenter. IPs from hosting networks behind a gateway that changes the exit IP per connection. Cheapest per request and usually the fastest. Works for the large majority of public pages. Weak spot: some big sites treat whole hosting ranges as suspicious.
• Residential. IPs that home internet providers assign to households. Sites tend to trust them more, so they help on targets that block datacenter ranges. They cost more and are slower per request, and most providers bill them per GB.
• Static (dedicated or ISP). One IP that stays yours. Right for logins, carts and anything with a session that must survive for hours. Wrong for volume crawling of one site, because every request lands on the same few addresses.
• Mobile. Carrier IPs shared by many phones. The most expensive type. Storm doesn’t sell mobile or ISP proxies, so if a target truly only answers mobile IPs, you’d need another provider for that site.
• Free public proxy lists. Slow, short-lived, often already flagged, and you can’t know who reads your traffic.

For a side-by-side of the rotation side, see rotating vs static proxies.

Most residential and many datacenter providers charge per gigabyte. That model punishes exactly what scrapers do: fetch lots of full HTML pages, plus retries. Heavy retail pages can run past a megabyte of HTML alone. At 2 MB a page, 500,000 pages a month is about 1 TB of traffic, and every failed attempt adds more.

Storm prices by concurrency instead. A 40-thread rotating plan is $39 a month whether that month moves 20 GB or 1 TB, and residential ports are priced per port, not per GB. So the question you answer when buying is “how many connections do I run at once”, not “how much data will I pull”. If you render pages in a browser, that matters even more, because a browser downloads scripts, fonts and images too. More on this in unlimited bandwidth proxies.

Thread math: threads = pages per second × seconds per page. One thread is one open connection at a time (explained in depth in proxy threads explained).

• 100,000 pages a day, plain HTTP, 3 s per page. 100,000 ÷ 86,400 = 1.2 pages/s × 3 = about 4 threads on average. The 40-thread plan ($39) covers it with room for retries and a second job. The 10-thread package ($14) is for testing, not for this.
• 1 million pages a day, plain HTTP, 4 s per page. 11.6 pages/s × 4 = about 47 threads around the clock. Take the 80-thread plan ($59); if you also scrape search engines from the same plan, go to 150 threads ($97) so the search share stays small.
• 20,000 pages a day rendered in a headless browser. Plan for around 10 threads per open tab. Four tabs in parallel means 40 threads, so the 40-thread plan is the floor and 80 threads gives breathing room.
• One strict marketplace, 30,000 pages a day. Put it on residential ports. 30,000 ÷ 86,400 × 300 s = about 104 requests per 5-minute IP window in total. Spread over 10 ports ($90), each IP carries about 10 requests per window, which is a gentle pace. With 1 port, one IP would carry all 104.

Residential sizing works differently: each port is one IP at a time, shared by up to 50 threads. More ports means more different IPs in parallel, which is what keeps per-IP request counts low on strict sites.

If the pilot shows blocks on datacenter IPs even at one request every few seconds, adding threads won’t help; it makes the pattern louder. That target belongs on residential ports, or needs a slower schedule.

If the job needs one identity for longer than 15 minutes (a logged-in dashboard, a B2B portal you have an account on), rotation works against you. Use a private dedicated proxy per account. And if the scraper runs on a platform whose outgoing IP changes every run, IP authorization can’t work there; dedicated proxies with user:pass can.

Proxies spread load; they don’t change a site’s rules. Read the site’s terms and robots.txt, stick to public pages, don’t scrape behind a login you aren’t allowed to automate, and don’t collect personal data you have no basis to hold. Keep per-domain concurrency modest even when your plan allows more: a 150-thread plan pointed at one small shop is a load problem for them and a ban for you. Storm doesn’t allow hacking, reselling, torrents or illegal activity, and outgoing email ports are blocked.