Home / Guides / Go (net/http & Colly)

How to use a proxy in Go (net/http and Colly)

Q: Does Go support SOCKS5 proxies?

Go’s Transport accepts socks5:// URLs, but you don’t need that here: Storm gateways are HTTP(S) proxies, not SOCKS. Use http://GATEWAY_IP:PORT , which covers both http and https sites.

Create an http.Transport with Proxy: http.ProxyURL(u), where u is http://GATEWAY_IP:PORT, and build your http.Client on it. In Colly, call c.SetProxy("http://GATEWAY_IP:PORT"). Then cap open connections at your plan’s thread count, because Go will happily open hundreds.

Updated October 2026Code & scraping frameworks

Go is a favourite for scrapers and monitoring jobs because goroutines are cheap. You can start 2,000 of them in a loop. That’s also the most common way Go users break their proxies: the default http.Transport has no limit on total connections, so a loop over a URL list opens far more tunnels than the plan allows, and the gateway starts refusing or resetting them. The second classic problem is keep-alive. Go reuses connections aggressively, so on a rotating gateway you can send 500 requests and see the same exit IP every time.

Below: plain net/http setup (fixed URL, environment variables, user:pass for dedicated proxies), a real concurrency cap, a fresh IP per request, the same in Colly, and the exact Go error strings with their fixes.

Which Storm plan fits

Go scrapers are fast and cheap on memory, so the limit you hit is threads, not CPU. A rotating plan fits most jobs: the Main gateway hands out a new IP per connection, and 40 threads ($39/mo) keeps 40 goroutines fetching at once with no per-GB bill. If your crawler logs in or must keep one IP for a few minutes, a residential port holds one IP for 5 minutes; a dedicated proxy is the pick only when the job needs the same static IP for weeks.

Rotating proxies (from $14/mo): 700,000+ IPs behind fixed gateway IP:PORTs. New IP on every request, or every 3 or 15 minutes. USA, EU, USA+EU or Worldwide. Unlimited bandwidth on every plan.

Get 40 threads for $39/mo See all rotating proxies plans

Before you start

Log in to the member area and copy your gateway IP:PORTs. They never change; the rotation happens on our side.
Add the public IP of the computer or server that will run your tool under Authorized IPs, click Save, and allow up to 15 minutes before testing. Rotating and residential proxies use IP authentication, so there is no username or password.
Dedicated proxies work with either IP authentication or a username and password. Use user:pass if your IP changes or the tool runs on several machines.
Count your threads: the tool’s total open connections must stay within your plan (for example 40 threads on the 40-thread plan).

On this page

Set up a proxy in Go, step by step
Copy-paste Go examples
Why MaxConnsPerHost doesn’t protect your thread limit
Keep-alive, connection reuse and when your IP changes
Colly specifics: SetProxyFunc, Async and LimitRule
Which gateway and how many threads for your Go job
Common errors and fixes
FAQ

Set up a proxy in Go, step by step

Authorize the machine that runs the binary
Rotating and residential gateways check the source IP. Run curl https://api.ipify.org on the box that will run your Go program, paste that IP under Authorized IPs in the member area, save, and give it up to 15 minutes. Serverless functions and hosted CI runners get a new IP each run, so run the binary on a fixed-IP server, or use dedicated proxies with user:pass.
Parse the gateway into a *url.URL
Use url.Parse("http://GATEWAY_IP:PORT"). Keep the http scheme even for https targets: Go opens a CONNECT tunnel through the gateway and does TLS with the website inside it. An https:// proxy URL tells Go to speak TLS to the proxy itself, which our gateways don’t do.
Build one Transport and one Client
Set Proxy: http.ProxyURL(proxyURL) on a new http.Transport and create one http.Client with a Timeout. Reuse it for the whole run; a client per request leaks idle connections.
Check the exit IP
GET https://api.ipify.org?format=json through the client. You should see a pool IP, and with keep-alives off on the Main gateway, a different one on each call.
Cap concurrency at your thread count
Start a fixed pool of worker goroutines (or use a buffered channel as a semaphore) sized to your plan: 40 on the 40-thread plan, less if other tools share the plan. Don’t rely on MaxConnsPerHost alone; the next section explains why it doesn’t cap the total.
Decide how the IP should change
For a new IP on every request, use the Main gateway and set DisableKeepAlives: true. For short sessions (a login followed by a few pages), keep keep-alives on and point the client at the 3-minute or 15-minute gateway, or at a residential port.

Copy-paste Go examples

Replace GATEWAY_IP:PORT with a gateway from your member area. Rotating and residential gateways need no credentials once your server IP is authorized. Only the dedicated example uses a username and password.

net/http: one gateway, sensible timeouts

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
	"time"
)

func main() {
	proxyURL, err := url.Parse("http://GATEWAY_IP:PORT") // http:// even for https sites
	if err != nil {
		panic(err)
	}

	tr := &http.Transport{
		Proxy:                 http.ProxyURL(proxyURL),
		ResponseHeaderTimeout: 30 * time.Second,
	}
	client := &http.Client{Transport: tr, Timeout: 60 * time.Second}

	resp, err := client.Get("https://api.ipify.org?format=json")
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	body, _ := io.ReadAll(resp.Body)
	fmt.Println(resp.StatusCode, string(body)) // a pool IP, not your server's
}

40 workers, new IP per request (Main gateway)

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
	"sync"
	"time"
)

const threads = 40 // your plan's thread count, minus anything else using it

func main() {
	proxyURL, _ := url.Parse("http://GATEWAY_IP:PORT")
	client := &http.Client{
		Timeout: 45 * time.Second,
		Transport: &http.Transport{
			Proxy:             http.ProxyURL(proxyURL),
			DisableKeepAlives: true, // one request per connection = fresh exit IP each time
		},
	}

	jobs := make(chan string)
	var wg sync.WaitGroup
	for w := 0; w < threads; w++ { // exactly `threads` goroutines touch the network
		wg.Add(1)
		go func() {
			defer wg.Done()
			for u := range jobs {
				resp, err := client.Get(u)
				if err != nil {
					fmt.Println("ERR", u, err)
					continue
				}
				io.Copy(io.Discard, resp.Body) // always drain and close the body
				resp.Body.Close()
				fmt.Println(resp.StatusCode, u)
			}
		}()
	}
	for i := 0; i < 500; i++ {
		jobs <- fmt.Sprintf("https://httpbin.org/anything/%d", i)
	}
	close(jobs)
	wg.Wait()
}

Dedicated proxy with username and password

proxyURL := &url.URL{
	Scheme: "http",
	Host:   "PROXY_IP:PORT",
	User:   url.UserPassword("USERNAME", "PASSWORD"), // escapes @ : / for you
}
tr := &http.Transport{Proxy: http.ProxyURL(proxyURL)}
client := &http.Client{Transport: tr, Timeout: 60 * time.Second}
// Go sends Proxy-Authorization on the CONNECT request automatically.

Proxy from environment variables

// export HTTPS_PROXY=http://GATEWAY_IP:PORT   (set before the program starts)
tr := &http.Transport{Proxy: http.ProxyFromEnvironment} // DefaultTransport does this too

Colly: round-robin over residential ports, 40 in flight

package main

import (
	"fmt"
	"time"

	"github.com/gocolly/colly/v2"
	"github.com/gocolly/colly/v2/proxy"
)

func main() {
	c := colly.NewCollector(colly.Async(true))
	c.SetRequestTimeout(45 * time.Second)

	// 5 residential ports = 5 exit IPs at once. For rotating, list your gateways.
	rp, err := proxy.RoundRobinProxySwitcher(
		"http://GATEWAY_IP:PORT1",
		"http://GATEWAY_IP:PORT2",
		"http://GATEWAY_IP:PORT3",
		"http://GATEWAY_IP:PORT4",
		"http://GATEWAY_IP:PORT5",
	)
	if err != nil {
		panic(err)
	}
	c.SetProxyFunc(rp) // Colly also turns keep-alives off here

	// One rule for every domain = one shared cap of 40 concurrent requests.
	c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 40, RandomDelay: 500 * time.Millisecond})

	c.OnResponse(func(r *colly.Response) {
		fmt.Println(r.StatusCode, r.Request.ProxyURL, r.Request.URL)
	})

	for i := 0; i < 200; i++ {
		c.Visit(fmt.Sprintf("https://httpbin.org/anything/%d", i))
	}
	c.Wait()
}

Why MaxConnsPerHost doesn’t protect your thread limit

MaxConnsPerHost sounds like the right knob, and many tutorials stop there. But Go counts it per connection key, and with a proxy that key is the proxy plus the target host and scheme. Crawl 30 different shops through one gateway with MaxConnsPerHost: 40 and Go may open up to 1,200 connections to that gateway, 40 for each shop. The same holds for MaxIdleConnsPerHost.

So treat MaxConnsPerHost as politeness toward a single website, and enforce the plan limit yourself:

a fixed pool of workers reading from a job channel (the second code example), a buffered channel used as a semaphore (make(chan struct{}, 40)), or errgroup with g.SetLimit(40).

Count everything that shares the plan. If a browser on the same plan is open, it can hold around 10 threads on its own, so your Go workers get the rest.

Keep-alive, connection reuse and when your IP changes

The Main rotating gateway picks the exit IP when a connection (or HTTPS tunnel) is opened. Every request sent down that connection afterwards leaves from the same IP. Go’s Transport keeps idle connections in a pool and reuses them, which is great for speed and bad if you expected a new IP per request.

New IP on every request: DisableKeepAlives: true on the Transport. It costs one extra handshake per request.
Same IP for a short session: don’t rely on connection reuse, which breaks at any time. Use the 3-minute or 15-minute gateway, which holds the IP for that window across connections.
Residential: every connection on a port leaves from that port’s current IP, and the IP changes every 5 minutes at fixed times. A connection that’s open at the change can drop, so give long downloads a retry.

Skip draining the body before Close() and Go can’t reuse the connection, so it opens new ones and wastes threads. Drain, close, and choose rotation on purpose. The backconnect proxy guide explains the gateway model.

Colly specifics: SetProxyFunc, Async and LimitRule

Three Colly details that the docs don’t spell out but that change how it behaves with a gateway:

SetProxy and SetProxyFunc switch keep-alives off. In Colly v2, setting a proxy also sets DisableKeepAlives = true on the collector’s Transport. With the Main gateway that gives you a new IP per request automatically. After c.WithTransport(...), set the proxy again.
RoundRobinProxySwitcher moves to the next URL on every request and records which proxy was used in r.Request.ProxyURL. That makes it easy to log which gateway or residential port produced a 403.
Parallelism belongs to a rule, not to a domain. A rule with DomainGlob: "*" shares one counter across every domain it matches, so it works as a global cap. Without colly.Async(true) Colly fetches one page at a time; with it, end with c.Wait().

A non-zero Delay makes Colly run one request at a time for that rule, so put strict delays on a narrow DomainGlob and keep the wide rule fast.

Which gateway and how many threads for your Go job

Size the plan from requests in flight, not page counts; bandwidth is unlimited.

Product pages, listings, public APIs: Main gateway with keep-alives off. The 40-thread plan ($39/mo) runs 40 workers; the 80-thread plan ($59) runs 80. Start at half your threads and raise it while the error rate stays flat.
Google and other search engines: Main gateway only, and no more than a quarter of your threads. On the 40-thread plan that’s 10 workers; on 150 threads, 37. The 429 guide covers backoff when the search engine pushes back.
Logins, carts, multi-step forms: the 3-minute gateway (USA, not for search engines) or the 15-minute gateway, with a cookiejar on the client so the session and the IP stay together.
Residential: each port allows up to 50 threads, all sharing one IP. Five ports ($50/mo) with 10 workers each gives 50 requests in flight spread over 5 IPs, which is usually kinder to a strict site than 50 on one port.

The 10-thread package ($14/mo) is meant for building and checking the scraper. See rotating proxy plans and residential ports for every size, or how threads are counted if the numbers don’t add up.

Common errors and fixes

read: connection reset by peer or EOF on every requestThe gateway accepted the TCP connection and dropped it because the machine’s public IP isn’t authorized yet, or was saved less than 15 minutes ago. Storm rotating and residential gateways reset unauthorized connections; they don’t send a 407. Compare curl https://api.ipify.org on the server with your Authorized IPs.

proxyconnect tcp: dial tcp GATEWAY_IP:PORT: connect: connection refusedNothing answered on that IP and port. Copy the gateway again from the member area and check outbound firewall rules on the server. Go adds the proxyconnect prefix whenever the problem is reaching the proxy itself.

tls: first record does not look like a TLS handshakeThe proxy URL starts with https://, so Go tried TLS with the gateway. Change it to http://GATEWAY_IP:PORT. The website still loads over HTTPS inside the tunnel.

Error text Proxy Authentication Required (or status 407)For https targets Go returns the CONNECT status text as the error. On dedicated proxies the username or password is wrong; build the URL with url.UserPassword so special characters are escaped. On rotating or residential gateways a 407 isn’t ours: check for HTTPS_PROXY pointing at another proxy.

context deadline exceeded (Client.Timeout exceeded while awaiting headers)Too many requests queued for your plan, or a slow site. Lower the worker count below your threads, make sure every body is closed, and raise ResponseHeaderTimeout for heavy pages.

The same exit IP on every requestKeep-alive reuse. Set DisableKeepAlives: true and use the Main gateway. With the 15-minute gateway or a residential port, a steady IP is expected.

Requests ignore HTTPS_PROXYThe variable was set after the program started (Go reads it once per process), the target is localhost (never proxied), or it matches NO_PROXY. A custom Transport without Proxy set ignores the environment entirely.

FAQ

Do Storm rotating proxies need a username and password in Go?

No. Rotating and residential gateways authorize the IP of your server, so the proxy URL is just http://GATEWAY_IP:PORT and no Proxy-Authorization header is needed. Only private dedicated proxies accept username and password, built with url.UserPassword.

How do I rotate proxies per request in Go?

With a backconnect gateway you don’t keep a list: point the client at the Main gateway and set DisableKeepAlives: true, and each request gets a new IP. To spread work over several gateways or residential ports, write a Proxy func that returns the next URL from a slice using an atomic counter, or use Colly’s RoundRobinProxySwitcher.

How many goroutines can I run on the 40-thread plan?

As many as you like, but only 40 should have a request open at the same moment, across every program and machine using that plan. Use 40 workers or a 40-slot semaphore. For Google, keep it to 10.

Does Go support SOCKS5 proxies?

Go’s Transport accepts socks5:// URLs, but you don’t need that here: Storm gateways are HTTP(S) proxies, not SOCKS. Use http://GATEWAY_IP:PORT, which covers both http and https sites.

Can I pay for a Go scraping plan with crypto?

Yes, manually. Contact support via the contact page with the package you want, send the USD price to the wallet address we reply with (BTC, ETH, TRC20 USDT or BNB/BEP20), then reply with the transaction ID and your username. We activate once we confirm the payment. Crypto isn’t recurring, so you pay again each month.

Still have questions? Contact us here. A real person answers.

Tool facts checked against the official documentation (October 2026): Go net/http package (Transport, ProxyURL, ProxyFromEnvironment) · Go net/url package (UserPassword) · Colly v2 package docs · Colly proxy package (RoundRobinProxySwitcher) · Colly docs: distributed scraping and proxy switchers. Storm Proxies facts: our plans page and refund policy.

Unlimited bandwidth. One flat monthly price.

Access is live the moment you pay, and the smallest package of each proxy type has a 24-hour money-back guarantee on your first order.

Get 40 threads for $39/mo Compare all plans