How to fix 429 Too Many Requests when using proxies
A 429 means the website, not the proxy, decided you sent too many requests in a short time. Slow down per site, wait as long as the Retry-After header says, retry with exponential backoff on a fresh connection, and spread the load over more IPs only if the site counts per IP. Rotation can’t fix limits tied to a session or account.
429 Too Many Requests is a status code a website sends when it thinks one client is asking for too much, too fast. RFC 6585 defines it and deliberately leaves out how a server identifies “a client” or counts requests. That’s why the same scraper can be fine on one site and get 429s within a minute on another.
Proxies change who the site thinks you are, but only partly. If the site counts per IP, spreading requests over more IPs keeps each one under the limit. If it counts per cookie, session or account, every IP you use still lands in the same bucket. And if your HTTP client keeps one connection open, a rotating gateway may send everything from one IP without you noticing.
This guide is about working within rate limits so your jobs finish reliably: reading what the site tells you, backing off correctly, and sizing threads and IPs per site. It isn’t about defeating limits that a site or API applies to your account.
Which Storm plan fits
When a site counts requests per IP, rotating proxies on the Main gateway help most: every new connection leaves from a different IP out of 700,000+, and a retry gets a fresh one. Residential ports and dedicated proxies give you a fixed number of IPs at once, so you control the rate by how many ports or proxies you spread work over. None of them change limits tied to your login or API key.
Rotating proxies (from $14/mo): 700,000+ IPs behind fixed gateway IP:PORTs. New IP on every request, or every 3 or 15 minutes. USA, EU, USA+EU or Worldwide. Unlimited bandwidth on every plan.
Get 40 threads for $39/mo See all rotating proxies plansBefore you start
- Log in to the member area and copy your gateway
IP:PORTs. They never change; the rotation happens on our side. - Add the public IP of the computer or server that will run your tool under Authorized IPs, click Save, and allow up to 15 minutes before testing. Rotating and residential proxies use IP authentication, so there is no username or password.
- Dedicated proxies work with either IP authentication or a username and password. Use user:pass if your IP changes or the tool runs on several machines.
- Count your threads: the tool’s total open connections must stay within your plan (for example 40 threads on the 40-thread plan).
Troubleshooting checklist, most likely cause first
- Read the 429 response, not just the code
Log the headers and body of the first 429.
Retry-Aftertells you how long to wait.RateLimit-*orX-RateLimit-*headers, if present, show the limit and when it resets. The body often says whether the limit is per IP, per account or per API key. That decides every step below. - Slow down per domain
Your plan’s threads are a ceiling, not a target. Cap concurrent requests per site separately from your total, and add a small delay between requests to the same domain. Ten workers spread over ten sites are very different from ten workers on one.
- Honor Retry-After, then back off exponentially
When the header is present, wait at least that long before you try that site again. When it isn’t, wait 1, 2, 4, 8 seconds and so on, with some random jitter so your workers don’t all retry at the same moment. Instant retries keep you inside the limit window and stretch the block.
- Make sure you’re really rotating
Clients that reuse connections (sessions, keep-alive pools, browsers) keep the same tunnel and so the same exit IP on the Main gateway. If every request looks like it comes from one IP, open a new connection per request or per retry. The Python requests guide shows how.
- Keep search engines to 25% of your threads
Search engines rate-limit hard. Use the Main gateway for them and at most a quarter of your threads: 10 on a 40-thread plan, 20 on an 80-thread plan. Don’t use the 3-minute gateway for search-engine scraping.
- Spread load over more IPs, if the limit is per IP
Residential: use more ports with fewer threads each, because all threads on a port share one IP. Dedicated: rotate requests across your proxies. Rotating: stay on the Main gateway. If you’ve done all this and still need more throughput, a bigger package adds capacity.
- Stop if the limit is on your account
If the 429 comes from an API with a key or from a logged-in account, the quota belongs to you, not to the IP. Use fewer calls, cache results, or ask the provider for a higher quota. Switching IPs won’t help and can break the service’s terms.
Handle 429 correctly in code
Replace GATEWAY_IP:PORT with a gateway from your member area. Each example waits for Retry-After when the site sends it and backs off when it doesn’t.
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
PROXY = "http://GATEWAY_IP:PORT"
retry = Retry(
total=5,
status_forcelist=[429, 503],
backoff_factor=1, # 1s, 2s, 4s, 8s ... when no Retry-After
respect_retry_after_header=True, # default; shown for clarity
allowed_methods=["GET", "HEAD"],
)
s = requests.Session()
s.proxies = {"http": PROXY, "https": PROXY}
s.mount("https://", HTTPAdapter(max_retries=retry))
s.mount("http://", HTTPAdapter(max_retries=retry))
print(s.get("https://httpbin.org/status/200", timeout=(10, 30)).status_code)import random, time, requests
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone
PROXY = "http://GATEWAY_IP:PORT" # Main gateway
proxies = {"http": PROXY, "https": PROXY}
def wait_seconds(resp, attempt):
ra = resp.headers.get("Retry-After")
if ra:
if ra.strip().isdigit(): # "120"
return int(ra)
when = parsedate_to_datetime(ra) # "Wed, 21 Oct 2026 07:28:00 GMT"
return max(0, (when - datetime.now(timezone.utc)).total_seconds())
return min(60, 2 ** attempt) + random.uniform(0, 1) # backoff + jitter
def fetch(url, tries=5):
for attempt in range(tries):
# new connection each time = new exit IP on the Main gateway
r = requests.get(url, proxies=proxies, timeout=(10, 30),
headers={"Connection": "close"})
if r.status_code != 429:
return r
time.sleep(wait_seconds(r, attempt))
return r# settings.py
CONCURRENT_REQUESTS = 40 # never above your plan's threads
CONCURRENT_REQUESTS_PER_DOMAIN = 8 # per-site cap
DOWNLOAD_DELAY = 0.5 # seconds between requests to one site
AUTOTHROTTLE_ENABLED = True # adapts the delay to server latency
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
RETRY_ENABLED = True
RETRY_TIMES = 4
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]What the site is counting
Rate limits are keyed to something the server can recognize. Knowing which one you’re hitting tells you whether proxies are part of the fix:
- Per IP. The most common for public pages. More IPs, and fewer requests per IP, solve it.
- Per session or cookie. The site counts requests carrying the same session cookie. Rotating IPs while sending one cookie jar changes nothing. Slow that session down or spread work over several independent sessions, each with its own consistent IP.
- Per account or API key. The quota is yours. Proxies don’t change it.
- Per fingerprint. Some sites group requests by headers and TLS characteristics. A thousand IPs sending the same unusual header set can still be treated as one heavy client. Use realistic, consistent headers per session.
- Global. The site is overloaded and throttles everyone. Often a 503 instead of a 429. Waiting is the only fix.
A quick test: after a 429, send one request through a fresh connection on the Main gateway with no cookies. If it succeeds, the limit was on the IP or session. If it fails the same way, look at account or global limits.
Retry-After and rate-limit headers
Retry-After is defined in RFC 9110 and comes in two forms: a number of seconds (Retry-After: 120) or an HTTP date (Retry-After: Wed, 21 Oct 2026 07:28:00 GMT). Your code has to handle both. urllib3’s Retry does this by default for 413, 429 and 503 responses.
Many APIs also send X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset, and an IETF draft standardizes RateLimit and RateLimit-Policy headers. They let you slow down before the 429. When remaining hits zero, pause until the reset time instead of pushing on.
Two mistakes to avoid. Retrying a 429 instantly on another IP ignores what the site asked for, and with a per-session limit lands you in the same penalty. And Scrapy’s built-in retry middleware retries 429s but doesn’t wait for Retry-After, so pair it with DOWNLOAD_DELAY or AutoThrottle (see the Scrapy guide).
Thread math: how many requests each IP really sends
What the site sees is requests per IP per minute, so work that out before you pick a plan or thread count:
- Rotating, Main gateway: each new connection takes an IP from the pool, so requests spread out by themselves, as long as your client opens new connections. Reused connections stay on one IP.
- Rotating, 3- and 15-minute gateways: the IP stays for minutes, so requests pile up on it. Good for sessions and browsing, poor for high-volume crawling of one site.
- Residential: one port is one IP for 5 minutes, and up to 50 threads share it. Fifty threads on one port all hit the site from one address. For a rate-limited site, 10 ports with 5 threads each is far gentler than 1 port with 50.
- Dedicated: each proxy is one fixed IP. With 10 proxies and a site that tolerates a request every few seconds per IP, your safe rate is roughly ten times that. Measure it; the site doesn’t publish the number.
The threads guide explains how threads, ports and connections relate.
When to add ports or proxies, and when not to
Add IPs when the limit is per IP, your per-IP rate is already moderate, and you still need more pages per hour. More residential ports or dedicated proxies raise the number of IPs working in parallel; a bigger rotating plan raises how many connections you can keep open across the pool.
Don’t add IPs when the 429s come from one login, one API key or one cookie jar, or when the site throttles everyone. In those cases, more IPs just means more failed requests. Cache what you’ve already fetched, skip pages you don’t need, and schedule heavy jobs for quieter hours.
Common errors and fixes
429 Too Many Requests with Retry-After: 60Wait 60 seconds before the next request to that site. Lower concurrency for that domain so it doesn’t happen again right after.urllib3.exceptions.MaxRetryError ... too many 429 error responsesYour Retry hit its total. Back off longer, lower concurrency, and check whether the limit is per account.Gave up retrying ... 429 Too Many RequestsRetries fired too fast. Enable AutoThrottle, add DOWNLOAD_DELAY and cut CONCURRENT_REQUESTS_PER_DOMAIN.FAQ
Is a 429 from the proxy or from the website?
From the website. A proxy problem looks like a 407, a connection error or a tunnel failure. A 429 means the request went through the proxy and the site answered it.
Do rotating proxies stop 429 errors?
They help when the site counts requests per IP and your client opens new connections. They don’t help with limits on your session, account or API key, and they don’t replace sensible request rates.
How many threads should I use for Google with Storm proxies?
At most 25% of your plan, on the Main gateway: 10 threads on the 40-thread plan, 20 on the 80-thread plan. Add delays between queries.
Can I trigger a new IP manually on a Storm residential port?
No. Residential ports change IP every 5 minutes at fixed times, and there is no manual rotation link. For a new IP per request, use the Main gateway of a rotating plan.
How long should I wait after a 429?
As long as Retry-After says. Without it, start at about a second and double the wait on each retry, with a random extra and a cap, for example one minute.
Does unlimited bandwidth mean I can send unlimited requests?
Bandwidth is unlimited on every Storm plan, so data volume never costs extra. The websites you visit still set their own request limits, and your plan sets how many connections you can keep open at once.
Still have questions? Contact us here. A real person answers.
Related guides
Tool facts checked against the official documentation (October 2026): RFC 6585: 429 Too Many Requests · RFC 9110: Retry-After · MDN: 429 Too Many Requests · IETF draft: RateLimit header fields · urllib3 Retry · Scrapy AutoThrottle · Scrapy RetryMiddleware. Storm Proxies facts: our plans page and refund policy.
Unlimited bandwidth. One flat monthly price.
Access is live the moment you pay, and the smallest package of each proxy type has a 24-hour money-back guarantee on your first order.