Home / Guides / Async Python (httpx, aiohttp)

How to use proxies with async Python (httpx and aiohttp)

In httpx, pass proxy="http://GATEWAY_IP:PORT" to httpx.AsyncClient; the old proxies= argument was removed in httpx 0.28. In aiohttp, pass proxy= on each request or on the ClientSession. Then wrap every request in an asyncio.Semaphore sized to your plan’s threads so you never open more connections than it allows.

Updated October 2026Code & scraping frameworks

Async Python lets one process keep hundreds of HTTP requests in flight while it waits on the network. The two libraries most people use for it are httpx (sync and async, with a requests-like API) and aiohttp (async only, older and very fast). Both route traffic through an HTTP proxy without extra packages.

Async code fails with proxies in its own ways. Copy-pasted examples still use httpx’s proxies= argument, which now raises a TypeError. asyncio.gather() over a big URL list starts everything at once and floods the proxy. And connection pooling, which is what makes async fast, also keeps you on one exit IP longer than you expect.

This guide covers the current httpx and aiohttp proxy syntax, how to cap concurrency at your thread count, when to reuse connections and when to force new ones for rotation, and the exceptions each library raises when something goes wrong.

Which Storm plan fits

Async scrapers push a lot of requests through few threads, which suits rotating proxies: the Main gateway gives each new connection a different IP, and per-thread pricing with unlimited bandwidth means you can fetch as much as the thread count allows. If many requests must share one IP for a few minutes, a residential port (up to 50 threads on one IP) or the 3/15-minute gateway fits. For code that runs on changing cloud IPs, use private dedicated proxies with a username and password.

Rotating proxies (from $14/mo): 700,000+ IPs behind fixed gateway IP:PORTs. New IP on every request, or every 3 or 15 minutes. USA, EU, USA+EU or Worldwide. Unlimited bandwidth on every plan.

Get 40 threads for $39/mo See all rotating proxies plans

Before you start

Log in to the member area and copy your gateway IP:PORTs. They never change; the rotation happens on our side.
Add the public IP of the computer or server that will run your tool under Authorized IPs, click Save, and allow up to 15 minutes before testing. Rotating and residential proxies use IP authentication, so there is no username or password.
Dedicated proxies work with either IP authentication or a username and password. Use user:pass if your IP changes or the tool runs on several machines.
Count your threads: the tool’s total open connections must stay within your plan (for example 40 threads on the 40-thread plan).

On this page

Set up async proxies, step by step
Copy-paste examples
httpx: proxy, mounts and the removed proxies argument
aiohttp: per-request proxies and its limits
Semaphore math: how many tasks per plan
Connection reuse vs rotation
Sync code, browsers and where to run it
Common errors and fixes
FAQ

Set up async proxies, step by step

Check your library versions
Run pip show httpx aiohttp. httpx 0.26 added the proxy argument and deprecated proxies; httpx 0.28 removed proxies entirely. If you’re on 0.28 or newer, every proxies= in your code must become proxy= or mounts=.
Authorize the machine that runs the script
Rotating and residential gateways accept only IPs saved under Authorized IPs. Add the public IP of the server or computer running the event loop, click Save, and allow up to 15 minutes.
Set the proxy on the client or session
httpx: httpx.AsyncClient(proxy=PROXY). aiohttp: session.get(url, proxy=PROXY), or set proxy= once on aiohttp.ClientSession. Use http://GATEWAY_IP:PORT as the value, also for HTTPS targets.
Size the Semaphore and the pool to your plan
Create asyncio.Semaphore(THREADS) and acquire it around each request. Match the connection pool too: httpx.Limits(max_connections=THREADS) or aiohttp.TCPConnector(limit=THREADS). Both libraries default to 100 connections, more than most plans allow.
Set timeouts
httpx: timeout=httpx.Timeout(30, connect=10). aiohttp: timeout=aiohttp.ClientTimeout(total=30, connect=10). A missing timeout lets a stuck connection hold a Semaphore slot forever.
Test the exit IP, then scale up
Fetch https://httpbin.org/ip a few times and confirm you see proxy IPs. Then run a small batch (50 URLs) and check the error rate before you start the full job.

Copy-paste examples

Replace GATEWAY_IP:PORT with your gateway. Rotating and residential gateways don’t take a username or password once your IP is authorized.

httpx AsyncClient with a proxy (httpx 0.26+)

import asyncio, httpx

PROXY = "http://GATEWAY_IP:PORT"     # http:// even for https sites

async def main():
    async with httpx.AsyncClient(proxy=PROXY, timeout=httpx.Timeout(30, connect=10)) as client:
        r = await client.get("https://httpbin.org/ip")
        print(r.json())

asyncio.run(main())

# OLD (httpx < 0.28 only):  httpx.AsyncClient(proxies={"all://": PROXY})
# NEW:                     httpx.AsyncClient(proxy=PROXY)

aiohttp: proxy per request or per session

import asyncio, aiohttp

PROXY = "http://GATEWAY_IP:PORT"

async def main():
    timeout = aiohttp.ClientTimeout(total=30, connect=10)
    async with aiohttp.ClientSession(timeout=timeout) as session:
        # per request: different requests can use different gateways
        async with session.get("https://httpbin.org/ip", proxy=PROXY) as r:
            print(await r.json())

    # or once for the whole session (recent aiohttp versions)
    async with aiohttp.ClientSession(proxy=PROXY, timeout=timeout) as session:
        async with session.get("https://httpbin.org/ip") as r:
            print(await r.json())

asyncio.run(main())

httpx: 40 concurrent requests with a Semaphore and retries

import asyncio, httpx

PROXY = "http://GATEWAY_IP:PORT"
THREADS = 40                                   # your plan's threads, or fewer

async def fetch(client, sem, url, tries=3):
    for attempt in range(tries):
        async with sem:                        # at most THREADS in flight
            try:
                r = await client.get(url)
                if r.status_code not in (429, 500, 502, 503, 504):
                    return url, r.status_code
            except httpx.TransportError:       # includes ProxyError and timeouts
                pass
        await asyncio.sleep(2 ** attempt)      # back off outside the semaphore
    return url, "failed"

async def main(urls):
    sem = asyncio.Semaphore(THREADS)
    limits = httpx.Limits(max_connections=THREADS, max_keepalive_connections=THREADS)
    async with httpx.AsyncClient(proxy=PROXY, limits=limits,
                                 timeout=httpx.Timeout(30, connect=10)) as client:
        return await asyncio.gather(*(fetch(client, sem, u) for u in urls))

urls = [f"https://httpbin.org/anything/{i}" for i in range(500)]
for url, status in asyncio.run(main(urls)):
    print(url, status)

aiohttp: new connection per request (new IP on the Main gateway)

import asyncio, aiohttp

PROXY = "http://GATEWAY_IP:PORT"
THREADS = 40

async def fetch(session, sem, url):
    async with sem:
        try:
            async with session.get(url, proxy=PROXY) as r:
                return url, r.status, (await r.text())[:60]
        except (aiohttp.ClientProxyConnectionError, aiohttp.ClientHttpProxyError,
                aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
            return url, type(e).__name__, ""

async def main(urls):
    sem = asyncio.Semaphore(THREADS)
    # force_close: no keep-alive, so each request opens a fresh tunnel
    conn = aiohttp.TCPConnector(limit=THREADS, force_close=True)
    timeout = aiohttp.ClientTimeout(total=30, connect=10)
    async with aiohttp.ClientSession(connector=conn, timeout=timeout) as session:
        return await asyncio.gather(*(fetch(session, sem, u) for u in urls))

print(asyncio.run(main(["https://api.ipify.org"] * 5)))

Dedicated proxy with username and password

from urllib.parse import quote
import httpx, aiohttp

user, pwd = "USERNAME", quote("PASSWORD", safe="")
AUTH_PROXY = f"http://{user}:{pwd}@PROXY_IP:PORT"

client = httpx.AsyncClient(proxy=AUTH_PROXY)          # httpx: credentials in the URL
# aiohttp: credentials in the URL work too
# async with session.get(url, proxy=AUTH_PROXY) as r: ...

httpx: proxy, mounts and the removed proxies argument

Since httpx 0.26 the supported way is a single proxy= string, which applies to both HTTP and HTTPS traffic. The older proxies= argument (a dict keyed like "all://" or "https://") was deprecated in 0.26 and removed in 0.28, so code written from older tutorials now fails with TypeError: AsyncClient.__init__() got an unexpected keyword argument 'proxies'.

When you need different routing per scheme or per domain, use mounts= with transports instead, for example mounts={"https://": httpx.AsyncHTTPTransport(proxy=PROXY)}. Mounts are also how you send one site through the 15-minute gateway and everything else through the Main one.

Watch out for environment variables: httpx reads HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and NO_PROXY by default (trust_env=True). If a proxy you didn’t configure keeps showing up, or a host bypasses the proxy, check those variables or pass trust_env=False.

aiohttp: per-request proxies and its limits

aiohttp takes the proxy on each request, which makes it easy to spread traffic across gateways: pass a different proxy= per call, or set a default on the session. Unlike httpx, aiohttp ignores proxy environment variables unless you create the session with trust_env=True.

Two details catch people. First, the proxy URL must be http://: aiohttp tunnels HTTPS through an HTTP proxy with CONNECT, while TLS to the proxy itself (an https:// proxy URL) has limited support. Second, TCPConnector defaults to limit=100 total connections and no per-host limit. Set limit to your thread count so the connector itself can’t exceed your plan even if your Semaphore code has a bug. aiohttp’s proxy_auth= argument is deprecated in recent releases; put dedicated-proxy credentials in the URL or in proxy_headers.

Semaphore math: how many tasks per plan

Your plan counts open connections at one moment. In async code, the number of tasks you create doesn’t matter; the number running a request right now does. That’s why the Semaphore wraps only the request, and the backoff sleep happens outside it, so waiting tasks don’t hold a slot.

Rotating, 40 threads: Semaphore(40) and a pool of 40. Two scripts on the same plan: 20 each.
Search engines: at most 25% of your threads, through the Main gateway. That is Semaphore(10) on a 40-thread plan and 37 on the 150-thread plan.
Residential: up to 50 threads per port, all sharing the port’s current IP. Use one Semaphore per port, and add ports when a site limits requests per IP.
Dedicated: up to 100 threads per proxy, always from the same IP. Spread load across proxies before one IP gets rate-limited.

For very large lists, a fixed set of worker tasks pulling from an asyncio.Queue uses less memory than gather() over a million coroutines, with the same concurrency cap.

Connection reuse vs rotation

Both libraries keep connections alive in a pool. For HTTPS, a kept-alive connection is one CONNECT tunnel through the gateway, and the Main rotating gateway assigns the exit IP per tunnel. So a pooled client sends many requests from the same IP, and a fresh connection gets a new one.

Pick on purpose. Reuse (the default) is faster, saves TLS handshakes and keeps one IP through a login or a paginated session. Rotation spreads requests over many IPs: in aiohttp use TCPConnector(force_close=True); in httpx set max_keepalive_connections=0 in httpx.Limits or send Connection: close. A middle path is one client per logical session: each account or crawl branch gets its own client, so each keeps its own IP. If you want an IP for minutes rather than per connection, use the 3- or 15-minute gateway.

Residential ports change IP at fixed times (minute 1, 6, 11, 16… of each hour), and connections open at that moment can drop. Treat ServerDisconnectedError, RemoteProtocolError and ReadError as retryable.

Sync code, browsers and where to run it

If you don’t need async, the Python requests guide covers the same setup with threads. Crawls with many domains and pipelines may be easier in Scrapy, which is async under the hood. For JavaScript-rendered pages use a browser such as Playwright.

Rotating and residential plans authorize by IP, so run async jobs on your machine or a VPS with a fixed IP. Hosted notebooks, serverless functions and many CI runners change their outgoing IP; use dedicated proxies with username and password there.

Common errors and fixes

TypeError: ... unexpected keyword argument 'proxies'httpx 0.28+ removed proxies. Use proxy="http://GATEWAY_IP:PORT", or mounts= for per-scheme routing.

httpx.ProxyError / aiohttp.ClientProxyConnectionErrorConnection refused means nothing answered on that IP and port: check for a typo or a firewall. If the error wraps a reset or disconnect instead, your public IP isn’t under Authorized IPs yet; save it and wait 15 minutes.

ClientHttpProxyError: 407 / httpx 407Wrong or unencoded credentials on a dedicated proxy. On rotating/residential an unauthorized IP shows as a disconnect, not a 407, so a 407 there comes from another proxy in the chain. See the 407 guide.

httpx.PoolTimeoutAll pooled connections are busy. Your Semaphore is larger than max_connections, or responses aren’t being closed. Match the two numbers and use async with for streamed responses.

ServerDisconnectedError / RemoteProtocolErrorA connection dropped. If it happens on every request from the first one, your IP isn’t authorized: the gateway accepts the connection and resets it. If it happens now and then, you have too many parallel connections for your plan, or a residential port rotated its IP mid-request. Lower concurrency and retry with backoff.

SSLError / TLS-in-TLS warningsThe proxy URL starts with https://. Use http://GATEWAY_IP:PORT; the site itself stays HTTPS.

Same IP for every requestThe pool is reusing one tunnel. Use force_close=True (aiohttp) or max_keepalive_connections=0 (httpx) on the Main gateway.

FAQ

Does httpx still support the proxies argument?

No. It was deprecated in httpx 0.26 and removed in 0.28. Use proxy= for one proxy, or mounts= with AsyncHTTPTransport(proxy=...) for routing rules.

Do I need a username and password for Storm rotating proxies in httpx or aiohttp?

No. Rotating and residential plans authorize the IP your script connects from, so the proxy is just http://GATEWAY_IP:PORT. Private dedicated proxies accept either IP authorization or user:pass in the URL.

How many concurrent requests can I run on a Storm plan?

As many as your plan has threads: 40 on the 40-thread rotating plan, up to 50 per residential port, up to 100 per dedicated proxy. Set the Semaphore and the connection pool to that number or lower.

Is aiohttp faster than httpx with proxies?

Both are limited mostly by the target site and your concurrency cap, not by the library. Pick httpx if you also want a sync API and HTTP/2, aiohttp if you want per-request proxies with a minimal API.

Why do I get the same IP on every request with asyncio?

Your client reuses kept-alive connections, and each tunnel keeps the IP it started with. Disable keep-alive for per-request rotation, or create one client per session when you want stable IPs per task.

Can I use a SOCKS proxy with httpx or aiohttp here?

Not with Storm. Our gateways are HTTP(S) only, so you don’t need httpx[socks] or aiohttp-socks.

Still have questions? Contact us here. A real person answers.

Tool facts checked against the official documentation (October 2026): HTTPX: Proxies · HTTPX changelog (0.26, 0.28) · HTTPX: Resource limits · HTTPX: Environment variables · aiohttp: Advanced client usage (proxies, connectors) · Python asyncio synchronization primitives. Storm Proxies facts: our plans page and refund policy.

Unlimited bandwidth. One flat monthly price.

Access is live the moment you pay, and the smallest package of each proxy type has a 24-hour money-back guarantee on your first order.

Get 40 threads for $39/mo Compare all plans