How to set up proxies for price monitoring
Run price checks through rotating proxies on the Main gateway, with a fixed concurrency budget per retailer rather than one big pool of workers. Read prices from the page’s structured data where it exists, schedule checks by how often each product changes, and move only the retailers that block datacenter IPs to residential ports.
A price monitor is a scraper with a deadline. You need today’s price for every SKU on your list, from every retailer you track, before someone makes a pricing decision. The usual failure isn’t that nothing works. It’s that the job works for most retailers and quietly fails on the strict ones: Newegg, Home Depot, Lowe’s, Mercado Livre and Amazon are the sites people most often tell us they get blocked or CAPTCHAs on. The second complaint is cost, especially on per-GB proxy plans, where checking 50,000 product pages several times a day adds up fast.
Proxies help by spreading your checks over many IPs so no single address hits a retailer’s rate limits. They don’t guarantee that a retailer won’t block you; big-box sites change their defenses often. What keeps a monitor reliable is the design around the proxies: a per-domain budget, pacing, clean parsing, retries that don’t make things worse, and knowing which retailers need residential IPs.
This guide walks through that design with Storm: product choice per retailer, worked sizing by SKU count, code for structured price extraction and per-domain limits, and the location and store pitfalls that make prices wrong even when the request succeeds.
Which Storm plan fits
Most price monitors fit rotating proxies: a new IP per connection, and a flat price per thread no matter how many product pages you load. 40 threads ($39) covers tens of thousands of SKU checks a day; 80 threads ($59) covers a few hundred thousand. For one or two retailers that keep blocking datacenter IPs, add a residential package (5 ports for $50) for those domains only, instead of moving the whole monitor.
Rotating proxies (from $14/mo): 700,000+ IPs behind fixed gateway IP:PORTs. New IP on every request, or every 3 or 15 minutes. USA, EU, USA+EU or Worldwide. Unlimited bandwidth on every plan.
Get 40 threads for $39/mo See all rotating proxies plansBefore you start
- Log in to the member area and copy your gateway
IP:PORTs. They never change; the rotation happens on our side. - Add the public IP of the computer or server that will run your tool under Authorized IPs, click Save, and allow up to 15 minutes before testing. Rotating and residential proxies use IP authentication, so there is no username or password.
- Count your threads: the tool’s total open connections must stay within your plan (for example 40 threads on the 40-thread plan).
Set up a price monitor with Storm, step by step
- List SKUs, retailers and how fresh each price must be
Group product URLs by retailer and give each group a refresh interval. Fast movers and promotions might need hourly checks; slow catalog items can be daily. This one list sets your request volume, and most monitors check far more often than they need to.
- Assign each retailer a product and a concurrency budget
Put every retailer on the rotating Main gateway by default, with a small per-domain limit such as 2 to 4 parallel requests. Retailers that block datacenter IPs in your pilot go to residential ports. Use USA gateways for US stores and EU for European ones.
- Authorize the monitoring server
Run the monitor on a server with a fixed IP, add that IP under Authorized IPs, save, and allow up to 15 minutes. Price-monitoring SaaS tools that fetch from their own cloud can’t use IP authorization; run your own collector or use dedicated proxies with a username and password.
- Extract prices from structured data first
Many product pages include schema.org
Productdata with anOfferthat holdsprice,priceCurrencyandavailability. Parsing that block survives layout changes far better than CSS selectors. Fall back to selectors only for retailers that don’t publish it. - Spread each run over its window
Don’t fire 20,000 checks at the top of the hour. Shuffle the URL list and spread it across the refresh window, so each retailer sees a steady, low rate. This alone often removes most CAPTCHAs.
- Validate every price before you store it
Reject zero, empty and wildly changed prices, and anything that came from a block or CAPTCHA page. Mark them for a retry later instead of saving bad data that someone acts on.
- Pilot, then scale
Run one full cycle at a low budget, check error and block rates per retailer, then raise budgets one retailer at a time. When you need more threads, upgrade in the member area; the unused part of the current plan is credited.
Copy-paste building blocks
Replace GATEWAY_IP:PORT with your gateways. The first snippet reads the price from JSON-LD; the second sets per-retailer limits in Scrapy so one strict site never takes the whole plan.
import json, re, requests
PROXY = "http://GATEWAY_IP:PORT"
LD_JSON = re.compile(r'<script[^>]+application/ld\+json[^>]*>(.*?)</script>', re.S | re.I)
def find_offer(node):
if isinstance(node, list):
for n in node:
hit = find_offer(n)
if hit: return hit
elif isinstance(node, dict):
if node.get("@type") in ("Offer", "AggregateOffer") and ("price" in node or "lowPrice" in node):
return node
for v in node.values():
hit = find_offer(v)
if hit: return hit
return None
def get_price(url):
r = requests.get(url, proxies={"http": PROXY, "https": PROXY}, timeout=(10, 30),
headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
"Accept-Language": "en-US,en;q=0.9"})
r.raise_for_status()
for block in LD_JSON.findall(r.text):
try:
offer = find_offer(json.loads(block))
except ValueError:
continue
if offer:
return {"url": url,
"price": offer.get("price") or offer.get("lowPrice"),
"currency": offer.get("priceCurrency"),
"stock": str(offer.get("availability", "")).rsplit("/", 1)[-1]}
return {"url": url, "price": None} # no structured data: use a selector fallback
print(get_price("https://www.example-shop.com/product/123"))CONCURRENT_REQUESTS = 36 # whole plan minus headroom
CONCURRENT_REQUESTS_PER_DOMAIN = 4 # default for retailers not listed below
DOWNLOAD_DELAY = 1
DOWNLOAD_SLOTS = { # strict retailers get a smaller, slower lane
"www.example-bigbox.com": {"concurrency": 2, "delay": 4},
"www.example-electronics.com": {"concurrency": 2, "delay": 3},
}
AUTOTHROTTLE_ENABLED = True # slows a domain down when it responds slower
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
RETRY_HTTP_CODES = [429, 500, 502, 503, 504]
RETRY_TIMES = 2Worked sizing by SKU count
Threads = checks per second × seconds per check. A plain HTTP product page through a rotating gateway usually takes a few seconds; use 4 seconds until you’ve measured your own.
- 5,000 SKUs on 3 retailers, twice a day, each run within 2 hours. 5,000 ÷ 7,200 s = 0.7 checks/s × 4 = about 3 threads. The 10-thread package ($14) is enough to prove it works; the 40-thread plan ($39) gives room to grow.
- 20,000 SKUs on 5 retailers, every 4 hours. 20,000 ÷ 14,400 = 1.4/s × 4 = about 6 threads. With per-domain budgets of 3 to 4, that’s still comfortably the 40-thread plan.
- 200,000 SKUs daily, finished within 6 hours. 200,000 ÷ 21,600 = 9.3/s × 4 = about 37 threads. Take the 80-thread plan ($59) rather than running a 40-thread plan at its limit.
- 50,000 SKUs hourly. 13.9/s × 4 = about 56 threads, plus retries: 150 threads ($97). Check first whether every SKU really needs hourly data.
- 10,000 SKUs a day on one strict retailer, on residential. Spread over the day that’s about 35 requests per 5-minute window. On 5 ports ($50), each IP carries about 7 per window.
Pages rendered in a headless browser cost far more: around 10 threads per open tab. Only render the retailers that need it. More on the math in proxy threads explained.
Per-domain budgets matter more than plan size
A 150-thread plan pointed at one retailer is 150 parallel requests at that retailer, which is a fast route to blocks. Your plan size is the total for all retailers together; each retailer should get its own small lane. Two to four parallel requests per retailer, with a delay, is a sensible start. Big marketplaces with many SKUs can take more if your pilot stays clean; strict big-box sites often need less.
Budgets also protect the rest of the monitor. When one retailer starts throwing CAPTCHAs, only its lane slows down, and every other price still arrives on time. For handling 429 responses and Retry-After headers, see fixing 429 Too Many Requests.
Why the price you collect can be wrong
- Store-based pricing. Home improvement and grocery chains often price by the store selected in the session. Set the store the way a shopper does, keep those cookies for the run, and record which store each price belongs to.
- Location. Storm gateways are USA, EU, USA+EU or Worldwide. There’s no city or state targeting, so regional prices come from the site’s own store or ZIP selector, not from the IP.
- Currency and language. EU stores may switch currency or language by IP or headers. Match the gateway to the store and send a consistent
Accept-Language. - Block pages that look normal. A CAPTCHA page with status 200 has no price in it. Treat “no price found” as a failed check, not as a price of zero.
- Variants and sellers. One URL can show different prices per size, color or marketplace seller. Decide which offer you track and store its identifier with the price.
Fewer requests, same freshness
The cheapest request is the one you don’t send. Many retailers publish XML sitemaps with lastmod dates; use them to find new and changed product URLs instead of re-crawling category pages. Where a server supports conditional requests, send If-None-Match or If-Modified-Since and a 304 Not Modified tells you nothing changed. Check category or search listing pages that show many prices at once before fetching each product page. And split SKUs into tiers by how often their price actually moves.
Storm doesn’t bill per GB, so these tricks don’t lower your bill directly. They lower the load on each retailer, which lowers your block rate, which is what keeps the monitor running.
Rotating, residential or dedicated for price monitoring
Rotating datacenter (Main gateway) is the default: independent product checks, a new IP per connection, and the lowest cost per check. Residential ports are for the retailers that keep blocking datacenter IPs in your pilot; keep only those domains on them. Private dedicated proxies fit monitors that log in to a B2B or wholesale portal to see your own trade prices, where the account should keep one IP. See which proxies fit web scraping for the general decision.
Common errors and fixes
FAQ
What proxies are best for price monitoring?
Rotating datacenter proxies for most retailers, because each check can come from a new IP and the cost per check is low. Residential ports for retailers that block datacenter IPs. Test your actual retailers first.
How many proxies do I need to monitor 10,000 products?
Very few threads if you spread the checks out: 10,000 checks over 2 hours at 4 seconds each is about 6 threads. The 40-thread Storm plan leaves room for retries and growth.
Will proxies stop retailers like Home Depot or Newegg from blocking me?
No proxy can promise that. Proxies spread your checks over many IPs; pacing, per-domain limits and realistic headers decide how often a retailer challenges you. Strict retailers may need residential ports and slower schedules.
Can I monitor prices for a specific city or store with Storm?
Not by IP: Storm offers USA, EU, USA+EU and Worldwide without city or state targeting. Select the store or ZIP on the retailer’s site inside your session instead.
Does checking prices every hour cost more on Storm?
No. Bandwidth is unlimited and you pay per thread or port. More frequent checks only need enough threads to finish each run on time.
Can I use one Storm plan for price monitoring and other scraping?
Yes. All tools share the same gateways and thread limit. Give each job its own share of threads so neither starves the other.
Still have questions? Contact us here. A real person answers.
Related guides
Tool facts checked against the official documentation (October 2026): schema.org: Offer · Scrapy docs: Settings (DOWNLOAD_SLOTS, CONCURRENT_REQUESTS_PER_DOMAIN) · Scrapy docs: AutoThrottle · RFC 9110: Conditional requests · Sitemaps protocol. Storm Proxies facts: our plans page and refund policy.
Unlimited bandwidth. One flat monthly price.
Access is live the moment you pay, and the smallest package of each proxy type has a 24-hour money-back guarantee on your first order.