How to collect web data for LLMs and RAG through proxies

Retrieval-augmented generation (RAG), fine-tuning sets and evaluation corpora all start the same way: fetch a lot of web pages, turn them into clean text or Markdown, split it into chunks and store it with its source. Tools like Crawl4AI and self-hosted Firecrawl were built for that last mile, returning LLM-ready Markdown instead of raw HTML, while Scrapy and requests still do the heavy lifting for large, static sites.

A proxy comes in once the crawl gets bigger than a few hundred pages. Sending every request from one IP gets you rate-limited, temporarily blocked or served degraded pages, and a corpus quietly full of “Access denied” and cookie-wall text is worse than a smaller clean one. Spreading requests across many IPs, at a polite pace, keeps the data you collect the data you meant to collect.

This guide covers the proxy settings for each tool, how many threads a browser-based crawler really uses, the rules that keep a crawl acceptable (robots.txt, site terms, rate limits), deduplication, and why billing per gigabyte is a poor match for rendered pages.

The thread limit on a rotating plan counts open connections at one moment. How far that goes depends entirely on the fetcher:

• requests, httpx, Scrapy: one connection per request in flight. A 40-thread plan runs about 40 parallel fetches, and a 150-thread plan about 150.
• Crawl4AI, Firecrawl’s Playwright service, any headless browser: a rendered page opens many connections for scripts, styles, images and fonts, around 10 per page. The same 40-thread plan renders about 4 pages at once.

That’s why the cheapest way to scale a corpus is to render only what needs rendering. Many “JavaScript” sites still ship the text in the HTML or in a JSON endpoint the page calls; fetch that with an HTTP client and save the browser for the rest. Crawl4AI’s text_mode=True on BrowserConfig skips images and other heavy content, which makes pages lighter, but keep your page count based on 10 threads per page anyway.

Several crawlers sharing one plan share its threads. If a Scrapy job uses 30 of 40 threads, a Crawl4AI job next to it has room for one rendered page.

A proxy changes where requests come from. It doesn’t change what a site allows. For a dataset you want to keep using, follow a few rules:

• robots.txt. Enable the robots check in your crawler. Many sites now list AI crawlers by name and disallow them; treat a disallow for your purpose as a no, even if your user agent string is different.
• Terms of service. Some sites forbid automated collection or reuse of their content for model training. Read them for any source you rely on, and prefer sources with an explicit licence, an official dump or a public endpoint.
• Per-domain pace. Concurrency should be spread across many domains, not aimed at one. A few requests at a time per site, with random delays and backoff on 429 and 503, is plenty for most sources.
• No logins, no personal data. Stay on public pages. Don’t collect behind accounts, and filter personal data out of the text before it reaches an index.

Storm proxies may not be used for hacking or other illegal activity, and that includes collecting data a site has closed to you.

Duplicates cost twice: once in embedding and storage, and again at query time, when the retriever returns five near-identical chunks and pushes useful context out of the prompt. Deduplicate at three levels:

• URL: strip tracking parameters and fragments, follow canonical tags, and keep a set of fetched URLs so a recrawl only fetches what’s new or changed.
• Content: hash the cleaned Markdown (as in the code above) to catch mirrors, print versions and paginated copies.
• Chunk: boilerplate such as cookie notices and footers repeats on every page; remove it before chunking, or hash chunks too.

Store fetched_at with every document. For refreshes, recrawl high-value sources on a schedule and compare hashes; unchanged pages don’t need new embeddings.

An LLM corpus is measured in pages, but most proxy plans are billed by data. A rendered page pulls in every script, font and image on it, so the bytes per page are many times the text you keep, and every retry, recrawl and refresh is billed again. Budgeting a big crawl on a per-GB plan means guessing page weight in advance and capping the crawler when the allowance runs out.

Storm rotating plans are priced by threads: 40 threads for $39, 80 for $59, 150 for $97 or 200 for $147 per month, with unlimited bandwidth. Recrawling the whole corpus every week costs the same as crawling it once. The number that matters is how many pages you fetch in parallel, which you control in the crawler settings.

Because rotating gateways authorize by IP, the crawler should live somewhere with a stable public IP: your own machine, an office server, or a small VPS. Hosted notebooks, serverless functions and many CI runners get a different IP each run, so the gateway won’t recognise them.

A practical layout is a VPS that runs the crawler on a schedule and writes Markdown plus metadata to storage, with embedding done wherever your vector database lives. If part of the pipeline must run in the cloud with a changing IP, give that part a dedicated proxy with username and password. If you also have AI agents browsing pages interactively, see the AI browser agent guide.