How to set up a proxy in Screaming Frog SEO Spider

Screaming Frog SEO Spider is the desktop crawler most SEOs use for technical audits: it follows links like a search engine bot and reports status codes, redirects, titles, canonicals, hreflang and much more. It runs from your own computer, so by default every request comes from your office or home IP.

A proxy helps in a few situations: a site’s firewall starts returning 403 or 429 halfway through a crawl, you need to see a site as visitors in the USA or the EU see it, you audit competitors and don’t want to burn your office IP, or your company network routes traffic through a proxy anyway. The setting itself takes a minute. What goes wrong is everything around it: forgetting the restart, more crawl threads than the proxy plan allows, JavaScript rendering that multiplies connections, and choosing a proxy type that doesn’t match a one-proxy tool.

This guide walks through the settings in current versions, the thread math for normal and rendered crawls, which Storm product fits which job, and the errors you’ll see when something is off.

The SEO Spider’s Max Threads is the number of URLs it fetches at the same time. Your rotating plan’s thread count is the number of connections that can be open through the gateway at the same time. Keep the first comfortably below the second, because the crawler also opens connections for robots.txt, redirects and, with rendering on, page resources.

• 10 threads ($14, test package): keep the default 5 crawl threads. Good for checking the setup, small for real audits.
• 40 threads ($39): up to about 30 crawl threads in text-only mode, or 3 to 4 with JavaScript rendering.
• 80 threads ($59): about 60 text-only, or 6 to 8 rendered.
• 150 threads ($97): about 110 text-only, or around 15 rendered.

More threads isn’t always faster. The official FAQ notes that the website itself usually sets the pace, and a small shared-hosting site will slow down or start returning 5xx errors long before you reach 30 threads. If something else uses the same plan at the same time (a scraper, a rank tracker, ScrapeBox), subtract its threads first.

With Configuration > Spider > Rendering set to JavaScript, the SEO Spider loads each page in headless Chrome, along with the scripts, stylesheets and images it references, waits for the AJAX timeout (5 seconds by default) and then reads the rendered HTML. A browser fetches those resources over several parallel connections, so one rendering thread behaves like a browser tab: plan on around 10 proxy threads for each one.

So a crawl that runs fine at 30 threads in text-only mode can produce timeouts and connection errors at the same setting with rendering switched on. Drop Max Threads when you enable rendering, or move to a bigger plan. Bandwidth isn’t a concern on Storm plans because it’s unlimited, which matters here: rendered crawls download far more data than raw HTML crawls.

Render only when you need to: for client-side frameworks, or to compare raw and rendered HTML. Most audit checks work in text-only mode at a fraction of the load.

Screaming Frog takes exactly one proxy server, and that shapes the choice.

• Rotating datacenter, Main gateway (best default): one address, a new exit IP per connection. A crawl’s requests spread over many IPs, which keeps any single IP well under a site’s per-IP rate limit. Pick the USA or EU location to match the market you audit, since many sites redirect or change content by visitor region.
• Rotating, 15-minute gateway: one IP for 15 minutes. Useful when a site ties a session or a consent cookie to the visitor IP and you want consistent results within a crawl section.
• Residential: one port is one IP at a time, changing every 5 minutes, with all threads on that port sharing it. In a one-proxy tool you crawl from a single household IP at a time, so keep threads low. It fits sites that refuse datacenter ranges, not big fast crawls.
• Private dedicated: one static IP that only you use. Best for crawling your own or a client’s site behind a firewall or CDN, because you can give that IP to the developers for their allowlist.

Skip the 3-minute gateway here: it’s a USA-only mix of IP types meant for account sign-ups, social sites and browsing, not for crawling. And keep in mind that Screaming Frog audits websites, not Google’s results pages. If you also scrape search engines with another tool, use the Main gateway and no more than 25% of your threads for that job.

For your own or a client’s site, a proxy is often the wrong fix for blocks. The official FAQ recommends asking the site owner to allowlist your IP and the Screaming Frog user agent when a firewall returns 403 or 429. A dedicated proxy makes that easy: one fixed IP, wherever you crawl from. Agree on a crawl rate with the developers before you run a big audit on a production server.

For competitor sites, stay polite. Keep Respect robots.txt on, set Max URI/s to a few URLs per second on smaller sites, and crawl outside the site’s peak hours. A rotating gateway spreads your requests across IPs, but your total load on their server is the same, so the rate limit you set is what makes a crawl responsible. A 429 is a signal to slow down, not to add threads.

Changing the user agent under Configuration > User-Agent to a browser string avoids blanket blocks on crawler names, but don’t impersonate Googlebot on sites you don’t own.