How to set up proxies for scraping Google search results
Use the Main rotating gateway, which gives a new IP on every connection, and keep Google traffic to no more than a quarter of your plan’s threads: 10 threads on a 40-thread plan. Render results in a headless browser, open a fresh connection per query, and slow down at the first CAPTCHA instead of retrying harder.
Google is the number one thing our customers scrape, and the top complaint is always the same: “I’m getting CAPTCHAs” or “the unusual traffic page”. Almost every time the cause is one of four things: too many parallel searches for the plan, a gateway that holds one IP for minutes, a scraper that reuses one connection for dozens of queries, or a plain HTTP client that no longer gets real results at all.
Google has also changed under scrapers twice recently. Since January 2025, search pages expect JavaScript, so simple HTTP requests often get a page asking to enable it instead of results. Since September 2025, the num=100 URL parameter no longer returns 100 results per page, so tracking the top 100 for a keyword now takes about ten page loads instead of one. Both changes push work into headless browsers and multiply request counts, and both change how many threads you need.
This guide covers the gateway choice, the thread rule, a working Playwright setup, honest sizing for real keyword counts, location limits, and what each Google error page means.
Which Storm plan fits
For Google, rotating proxies through the Main gateway are the right Storm product: each query can leave from a different IP, which is what Google’s per-IP limits reward. Size the plan so search uses a quarter of it: 40 threads ($39) gives 10 search threads, enough for roughly one browser tab and a few hundred keywords a day; 150 threads ($97) gives about 37. Residential ports are a poor fit for volume SERP work, because every thread on a port shares one IP for 5 minutes.
Rotating proxies (from $14/mo): 700,000+ IPs behind fixed gateway IP:PORTs. New IP on every request, or every 3 or 15 minutes. USA, EU, USA+EU or Worldwide. Unlimited bandwidth on every plan.
Get 40 threads for $39/mo See all rotating proxies plansBefore you start
- Log in to the member area and copy your gateway
IP:PORTs. They never change; the rotation happens on our side. - Add the public IP of the computer or server that will run your tool under Authorized IPs, click Save, and allow up to 15 minutes before testing. Rotating and residential proxies use IP authentication, so there is no username or password.
- Count your threads: the tool’s total open connections must stay within your plan (for example 40 threads on the 40-thread plan).
Set up a Google SERP scraper with Storm
- Choose rotating proxies and the Main gateway
In the member area, copy a Main gateway
IP:PORTfrom your rotating plan. Main picks a new exit IP for each new connection. Don’t use the 3-minute gateway for search engines: it’s a USA-only mix meant for sign-ups, social sites and browsing, and holding one IP for minutes is exactly what triggers Google’s limits. - Pick the location that matches the market
Use USA for US results, EU for European results, USA+EU for a mix, or Worldwide for scraping across markets. There is no city or state targeting, so set country and language with Google’s own
glandhlURL parameters, for example&gl=us&hl=en. - Authorize your server’s IP
Add the fixed public IP of the machine that runs the scraper under Authorized IPs, save, and wait up to 15 minutes. Rank trackers that run in a vendor’s cloud can’t use IP authorization; run your own on a VPS instead.
- Budget a quarter of your threads for Google
40-thread plan: 10. 80 threads: 20. 150 threads: about 37. 200 threads: 50. The remaining threads stay free for other sites and tools on the same plan. A browser tab uses around 10 connections, so 10 search threads means about one tab at a time.
- Render in a browser and isolate each query
Load results in a headless browser, with a fresh browser context per keyword so each query opens new connections and gets a new IP. Block images, fonts and media to cut the work per page. The code below does this in Playwright for Python.
- Detect block pages and back off
If the page URL contains
/sorry/or the text mentions unusual traffic, stop that worker, wait, and retry later on a new connection. Retrying instantly at the same speed only spreads the block to more IPs. - Run a small batch, then scale with the plan
Start with 50 keywords at one tab. If CAPTCHAs stay rare, add a tab. If you need more tabs than your search budget allows, move up a plan in the member area rather than raising the share.
Playwright SERP collector (Python)
Two parallel tabs fit the 20 search threads of an 80-thread plan; use 1 on a 40-thread plan. Replace GATEWAY_IP:PORT with your Main gateway. Google changes its markup often, so treat the result selector as a starting point.
import asyncio, random
from urllib.parse import quote_plus
from playwright.async_api import async_playwright
PROXY = {"server": "http://GATEWAY_IP:PORT"} # Main gateway, IP-authorized
TABS = 2 # ~10 threads per tab; stay within 25% of the plan
KEYWORDS = ["proxy setup guide", "rotating proxies python"]
PAGES = 3 # 10 results per page, so 3 pages = top 30
async def block_heavy(route):
if route.request.resource_type in ("image", "font", "media"):
await route.abort()
else:
await route.continue_()
async def search(browser, sem, kw):
async with sem:
results = []
for page_no in range(PAGES):
ctx = await browser.new_context(proxy=PROXY, locale="en-US") # new connections = new IP
page = await ctx.new_page()
await page.route("**/*", block_heavy)
url = (f"https://www.google.com/search?q={quote_plus(kw)}"
f"&gl=us&hl=en&start={page_no * 10}")
try:
await page.goto(url, wait_until="domcontentloaded", timeout=45000)
if "/sorry/" in page.url or "unusual traffic" in (await page.content()).lower():
print("blocked:", kw, "page", page_no + 1, "- backing off")
await asyncio.sleep(random.uniform(60, 120))
break
links = await page.eval_on_selector_all(
"a:has(h3)", "els => els.map(e => e.href)")
results += links
finally:
await ctx.close()
await asyncio.sleep(random.uniform(4, 9)) # human-ish pacing between pages
print(kw, len(results), results[:3])
async def main():
sem = asyncio.Semaphore(TABS)
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
await asyncio.gather(*(search(browser, sem, kw) for kw in KEYWORDS))
await browser.close()
asyncio.run(main())Why the Main gateway and the 25% rule
Google limits how much traffic it accepts from one IP in a short window. When a single address sends searches too fast, it gets the “unusual traffic” page and a CAPTCHA. The Main gateway attacks that directly: each new connection exits from a different IP in a 700,000+ pool, so no single address carries a run of queries.
The quarter-of-your-threads rule is the other half. It caps how hard you push search engines from one plan at a time, which keeps the rate per IP low and leaves the rest of the plan for other work. Going above it doesn’t buy more results; it buys more CAPTCHAs, and every CAPTCHA page you load costs a slot that could have fetched a real result.
The timed gateways are the wrong tool here. The 3-minute and 15-minute gateways hold one IP on purpose, so twenty queries in a row would all come from one address.
Keywords per day: worked sizing
Since num=100 stopped working, plan for 10 results per page load. Top 30 = 3 loads per keyword; top 100 = 10 loads. In a headless browser with images blocked and pacing between pages, budget about 8 to 10 seconds per load and about 10 threads per tab.
- 300 keywords, top 30, daily. 900 loads × 9 s ≈ 2.3 hours on one tab. One tab = 10 search threads = the 40-thread plan ($39).
- 2,000 keywords, top 30, daily. 6,000 loads × 9 s = 15 tab-hours. Two tabs finish in about 7.5 hours: 20 search threads, so the 80-thread plan ($59).
- 5,000 keywords, top 50, daily. 25,000 loads × 9 s ≈ 62 tab-hours, so you need 3 tabs running around the clock: about 30 to 37 search threads, which is the 150-thread plan ($97).
- Top 100 for 1,000 keywords. 10,000 loads ≈ 25 tab-hours: 2 tabs for about half a day, so 20 search threads on the 80-thread plan.
If your tracker only needs the top 10, each keyword is a single load and the numbers drop by two thirds or more. Tracking fewer positions is often the cheapest optimization there is. Harvesting with a desktop tool follows the same rule; see the ScrapeBox setup.
Plain HTTP vs a headless browser
Since January 2025, Google expects search pages to run JavaScript. Plain HTTP clients such as requests, cURL or Scrapy without a browser often receive a page asking to enable JavaScript instead of results. That’s why the setup above uses a headless browser. It costs more threads per query, so size for it rather than hoping the old HTTP approach still holds; if you try it, verify that responses contain real result links, not a JavaScript notice.
A browser also gives Google a consistent picture: a real rendering engine, a normal Accept-Language, and cookies inside one context. Keep one context per keyword and close it afterwards so cookies from one search don’t follow the next. Setup details for each library are in the Playwright and Puppeteer guides.
Google Maps and local results
Maps is the heaviest Google page you can load in a browser: map tiles, scripts and continuous background requests. One Maps tab can easily use more than the 10 connections a search tab needs, and on a small plan those connections queue up, which feels like “the proxies are slow”. Block images, scroll the result list instead of panning the map, run fewer Maps tabs, and give Maps its own share of the plan.
Local results also depend on location. Storm offers USA, EU, USA+EU and Worldwide, without city or state targeting, so you can’t make Google think a query comes from a specific city by IP. Storm can’t give you local packs for a specific city, so that part of the job needs a different tool; country-level results work fine with gl and hl.
Residential ports for Google: when they don’t fit
A residential port is one IP at a time that changes every 5 minutes, and up to 50 threads share it. For Google that means every query on the port in a 5-minute window comes from the same household address, which is the pattern you want to avoid. Residential ports are useful for many strict sites, but for volume SERP collection the rotating Main gateway spreads queries far wider for less money.
Google’s official alternative, the Custom Search JSON API, is closed to new customers and is being discontinued for existing ones on January 1, 2027, so it isn’t a plan B to build on.
Common errors and fixes
google.com/sorry/index with a CAPTCHAToo many queries from one IP or one connection. Check you’re on the Main gateway, that each query opens a new context or connection, and that search uses no more than a quarter of your threads. Back off for a minute or two before retrying.num=100 returns only 10 resultsGoogle no longer honors the parameter. Page with start=0, 10, 20… and size your plan for the extra loads.gl and hl in the URL.FAQ
What are the best proxies for scraping Google?
Proxies that give a different IP on every query, with enough pool size that no address repeats often. On Storm that’s the rotating Main gateway. Static IPs and long sticky sessions get rate-limited quickly on Google.
How many threads should I use for Google on a Storm plan?
At most a quarter of your threads: 10 on the 40-thread plan, 20 on 80, about 37 on 150, 50 on 200. With browser rendering, that is roughly one tab per 10 threads.
Can I use the Storm 3-minute gateway for Google?
No. It’s meant for account registration, social sites and browsing, and it’s USA only. For search engines use the Main gateway.
Do proxies stop Google CAPTCHAs completely?
No. They spread queries over many IPs, which lowers how often you hit Google’s limits. Pace, concurrency, connection reuse and how realistic your browser looks still decide how many CAPTCHAs you see.
Can I scrape Google results for a specific city?
Not by IP with Storm: locations are USA, EU, USA+EU and Worldwide, with no city or state targeting. Country and language can be set with the gl and hl parameters.
Why did my rank tracker suddenly need more requests?
Since September 2025 Google ignores num=100, so 100 results take about ten page loads instead of one. Plan threads for the larger count, or track fewer positions.
Still have questions? Contact us here. A real person answers.
Related guides
Tool facts checked against the official documentation (October 2026): Playwright docs: Network (proxy and route) · Google Search Help: Unusual traffic from your computer network · Google: Custom Search JSON API overview. Storm Proxies facts: our plans page and refund policy.
Unlimited bandwidth. One flat monthly price.
Access is live the moment you pay, and the smallest package of each proxy type has a 24-hour money-back guarantee on your first order.