Proxy & Proxy Pool

Whether you are traversing a corporate network, rotating egress IPs for a scraping job, or evading IP blocks on a target site, proxies are a frequent need for an HTTP client. HTTPC ships four proxy modes — single proxy, system-proxy detection, proxy-pool rotation, and status-code-triggered rotation — covering the full spectrum from "fixed egress" to "a different IP per request", and they cooperate with SSRF protection, TLS validation, and the retry engine. All proxy configuration lives in ConnectionConfig.

Proxy Mode Overview

The four modes take effect automatically by priority; when several are configured, only the highest priority applies:

Priority	Setting	Behavior	Typical scenario
1 (highest)	`ProxyURL`	Always use the specified proxy (single-proxy mode)	Corporate egress, local VPN port
2	`ProxyPool`	Rotate across the pool, with circuit breaking and recovery	Scraping, load dispersion, IP rotation
3	`EnableSystemProxy`	Auto-detect system proxy settings	Desktop apps following user configuration
4 (lowest)	None	Direct connection	Default behavior

TIP

If both ProxyURL and ProxyPool are set, ProxyURL wins. To use the proxy pool, clear ProxyURL.

Single Proxy Configuration

ProxyURL designates one fixed proxy and supports four protocols:

Protocol	Form	Notes
HTTP	`http://proxy:8080`	Most common; HTTPS requests are forwarded through a CONNECT tunnel
HTTPS	`https://proxy:8443`	TLS to the proxy server itself
SOCKS5	`socks5://proxy:1080`	Resolves the target domain locally, then connects via the proxy
SOCKS5h	`socks5h://proxy:1080`	Domain resolution is delegated to the proxy, avoiding local DNS pollution

package main

import (
    "log"

    "github.com/cybergodev/httpc"
)

func main() {
    cfg := httpc.DefaultConfig()
    cfg.Connection.ProxyURL = "socks5://proxy.example.com:1080"

    client, err := httpc.New(cfg)
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    result, err := client.Get("https://api.example.com/data")
    if err != nil {
        log.Fatal(err)
    }
    // Output: status: 200, proxy: socks5://proxy.example.com:1080
    log.Printf("status: %d, proxy: %s", result.StatusCode(), result.Meta.ProxyURL)
}

Authentication and Masking

Proxy credentials go directly into the userinfo part of the URL:

cfg := httpc.DefaultConfig()
cfg.Connection.ProxyURL = "http://user:[email protected]:8080"

Credentials are masked automatically

Config.String() replaces the username and password in proxy URLs with ***:***; URLs in error messages and logs are sanitized the same way (credentials and sensitive query parameters are masked). Credentials never leak into logs, but the configuration itself still needs safekeeping.

System Proxy Detection and NO_PROXY

Once enabled, the operating system's proxy settings are detected automatically — no need to set ProxyURL by hand:

cfg := httpc.DefaultConfig()
cfg.Connection.EnableSystemProxy = true

Platform Differences

Platform	Detection source
Windows	Registry Internet Settings (`ProxyEnable` / `ProxyServer`)
macOS	The `networksetup` command reading the preferred network service's Web/Secure Web Proxy
Linux	Environment variables `HTTP_PROXY` / `HTTPS_PROXY`

Meta.ProxyURL does not cover the system proxy

The system-proxy selection is not recorded in Result.Meta.ProxyURL (the field only has a value under an explicit Connection.ProxyURL or ProxyPool; it stays empty for direct connections and system proxies). To confirm the egress proxy on a per-request basis, use an explicit configuration instead.

Detection Order and Details

Environment variables first (all platforms): HTTP_PROXY / HTTPS_PROXY / NO_PROXY are read first (both cases recognized); if any is set, it is used directly without consulting system settings.
Platform detection as fallback: without environment variables, platform settings are read. On Linux desktops (GNOME/KDE) the proxy is usually already exported to environment variables by the session, so the engine does not read gsettings/dconf directly.
Request routing: HTTPS requests prefer HTTPS_PROXY and fall back to HTTP_PROXY when unset; HTTP requests use only HTTP_PROXY, also falling back to HTTPS_PROXY when unset — the same resolution order as net/http.
CGI environments connect directly: a proxy from environment variables does not take effect in CGI environments (REQUEST_METHOD set), matching net/http behavior.
Bare addresses are auto-completed: values without a protocol prefix such as HTTP_PROXY=proxy:8080 are treated as http://.
Caching: the detection result is cached for the client's lifetime and does not hot-reload when environment variables change; create a new client to pick up changes.

NO_PROXY Bypass Rules

NO_PROXY lists hosts that bypass the proxy, with semantics matching net/http's httpproxy package:

Rule	Example	Match scope
Bypass everything	`*`	All hosts connect directly
Domain suffix	`example.com` or `.example.com`	The domain and all its subdomains
Wildcard subdomains	`*.example.com`	Equivalent to `.example.com`
IP literal	`10.0.0.5`	Exact match on that IP
CIDR range	`10.0.0.0/8`	All IPs inside the range
Host + port	`example.com:443`	Host matches and the port is exact

Separate multiple rules with commas; localhost always connects directly and does not need to be listed in NO_PROXY.

bash

# Linux/macOS example
export HTTPS_PROXY=http://proxy.corp.example.com:8080
export NO_PROXY=localhost,127.0.0.1,.internal.corp.com,10.0.0.0/8

powershell

# Windows (PowerShell) example
$env:HTTPS_PROXY = "http://proxy.corp.example.com:8080"
$env:NO_PROXY = "localhost,127.0.0.1,.internal.corp.com"

Limitation of dynamic localhost proxies

In system-proxy mode, the SSRF exemption list is probed only once, at client construction. If the system proxy switches to a new internal/loopback address at runtime (e.g. 127.0.0.1), that address may be blocked by SSRF protection. For dynamic setups, set Connection.ProxyURL explicitly (proxy addresses are always exempt from SSRF validation) or configure SSRFExemptCIDRs.

Proxy Pool

When requests need to be spread across multiple proxy IPs (scraping, load dispersion, IP rotation), the proxy pool provides automatic rotation, passive circuit breaking, and status-based switching — with no external components.

Basic Usage

package main

import (
    "log"

    "github.com/cybergodev/httpc"
)

func main() {
    cfg := httpc.DefaultConfig()
    cfg.Connection.ProxyPool = []string{
        "http://proxy1:8080",
        "http://proxy2:8080",
        "http://proxy3:8080",
    }
    cfg.Connection.ProxyPoolStrategy = httpc.ProxyStrategyRoundRobin // default, may be omitted

    client, err := httpc.New(cfg)
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    result, err := client.Get("https://api.example.com/data")
    if err != nil {
        log.Fatal(err)
    }
    // Output: status: 200, proxy: http://proxy1:8080 (the next request automatically lands on proxy2)
    log.Printf("status: %d, proxy: %s", result.StatusCode(), result.Meta.ProxyURL)
}

Pool entries support the http, https, socks5, and socks5h protocols and can be mixed.

Configuration Fields

Field	Type	Default	Description
`ProxyPool`	`[]string`	`nil`	List of proxy URLs
`ProxyPoolStrategy`	`ProxyStrategy`	`ProxyStrategyRoundRobin`	Selection strategy
`ProxyFailureThreshold`	`int`	`3` (0 falls back)	Consecutive-connection-failure threshold for circuit breaking
`ProxyCooldown`	`time.Duration`	`30s` (0 falls back)	Cooldown for circuit-broken proxies
`ProxyRotatePerRequest`	`bool`	`false`	Force a different proxy for each independent request (disables idle connection reuse)
`ProxyRotateOnStatus`	`[]int`	`nil`	Status codes that trigger a proxy-switching retry (each must be 100–599)

Selection Strategy

Strategy	Constant	Description
Round-robin (default)	`ProxyStrategyRoundRobin`	Cycles through proxies in order; every selection advances the cursor
Random	`ProxyStrategyRandom`	Picks uniformly at random from healthy proxies

Round-robin + retries = automatic IP switching

The round-robin strategy advances the cursor on every selection, so a retry that triggers another selection naturally lands on the next proxy — no extra configuration needed.

Passive Circuit Breaking

The proxy pool has built-in passive health checking. Only connection-level failures (dial/TLS) trigger circuit breaking; HTTP status codes do not:

text

Proxy connection failure
    ↓
Failure count +1
    ↓
Consecutive failures ≥ ProxyFailureThreshold → circuit open (removed from rotation)
    ↓
Wait ProxyCooldown → half-open probe (restored to rotation)
    ↓
Success → reset count, close circuit
First failure → re-open circuit

cfg.Connection.ProxyFailureThreshold = 5        // more tolerant of transient flaps
cfg.Connection.ProxyCooldown = 60 * time.Second // longer cooldown

When every proxy is circuit-broken, the proxy with the shortest cooldown (closest to recovery) is returned as a fallback instead of failing outright.

Status-Code Rotation

For Cloudflare/WAF-style IP blocking — when the response carries specific status codes, the proxy is switched automatically and the request retried:

package main

import (
    "log"

    "github.com/cybergodev/httpc"
)

func main() {
    cfg := httpc.DefaultConfig()
    cfg.Connection.ProxyPool = []string{
        "http://proxy1:8080",
        "http://proxy2:8080",
        "http://proxy3:8080",
    }
    cfg.Connection.ProxyRotateOnStatus = []int{403} // switch proxy and retry on 403
    cfg.Retry.MaxRetries = 3                        // retries must be enabled

    client, err := httpc.New(cfg)
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    result, err := client.Get("https://protected-site.example.com/data")
    if err != nil {
        log.Fatal(err)
    }
    // Output: status: 200, proxy: http://proxy2:8080, attempts: 2
    log.Printf("status: %d, proxy: %s, attempts: %d",
        result.StatusCode(), result.Meta.ProxyURL, result.Meta.Attempts)
}

Status-code rotation ≠ circuit breaking

Rotation triggered by ProxyRotateOnStatus does not circuit-break the proxy — IP blocks are often target-specific (a proxy blocked on site A may work fine on site B). Circuit breaking is triggered only by connection-level failures. Retry.MaxRetries > 0 is required for it to take effect.

When ProxyRotateOnStatus is set and the pool holds multiple proxies, the retry budget is automatically raised to len(ProxyPool) - 1 (capped at the MaxRetries limit of 10), ensuring every proxy gets a chance.

Per-Request Rotation

ProxyRotatePerRequest solves the problem of connection reuse pinning the proxy tunnel: an HTTP connection pool reuses established TCP connections, including their proxy tunnels. That means consecutive requests to the same host reuse the previous request's proxy, even when ProxyPoolStrategy has already rotated the selector cursor.

When enabled, all idle connections are closed at the start of each request, forcing the Transport to re-evaluate the proxy pool — the cost is no connection reuse (each request establishes a new connection + proxy tunnel), but per-request rotation is guaranteed:

package main

import (
    "log"

    "github.com/cybergodev/httpc"
)

func main() {
    cfg := httpc.DefaultConfig()
    cfg.Connection.ProxyPool = []string{
        "http://proxy1:8080",
        "http://proxy2:8080",
        "http://proxy3:8080",
    }
    cfg.Connection.ProxyRotatePerRequest = true // rotate the proxy on every request

    client, err := httpc.New(cfg)
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    for i := 0; i < 3; i++ {
        result, err := client.Get("https://api.example.com/data")
        if err != nil {
            log.Fatal(err)
        }
        // Output (in order): http://proxy1:8080 / http://proxy2:8080 / http://proxy3:8080
        log.Printf("request %d went through: %s", i+1, result.Meta.ProxyURL)
    }
}

When to use it

Suited for scraping the same host — every request comes from a different source IP, lowering the risk of being IP-blocked by the target site. For requests to different hosts, connection reuse does not pin the same proxy, so enabling it is usually unnecessary.

Like ProxyRotateOnStatus, ProxyRotatePerRequest automatically raises the retry budget to len(ProxyPool) - 1 when the pool holds multiple proxies (capped at 10), ensuring every proxy is tried at least once.

Deterministic Rotation

Status-code rotation retries differ from ordinary retries: the engine reserves a base proxy index per request, and the Nth retry deterministically uses the proxy at "base + N", which yields three guarantees:

Every retry switches proxy: within one request, retry N and retry N-1 always land on different proxies;
Redirect chains stay on track: multiple hops within one attempt share the same index, so following redirects never desynchronizes the proxy selection by consuming extra picks;
Rotation continues across requests: the base index increments across requests, preserving the overall round-robin/random distribution.

In addition, connection failures of the proxy itself (dial/TLS failures) are always retryable while rotation is active — even when the regular classification says non-retryable (e.g. a permanent address error), the next attempt switches proxies and tries again, and passive circuit breaking naturally weeds out bad proxies. For the retry-side details, see Retry and Fault Tolerance.

Finding Out Which Proxy a Request Used

Proxy-pool setups often need to audit "which egress this request actually used". Two tools work together:

Result.Meta.ProxyURL: reports the proxy used by the attempt that produced the final response; an empty string for direct connections, and system proxies (EnableSystemProxy) are likewise not recorded (still empty). Under rotation each retry may use a different proxy — this field reflects the final attempt.
The WithOnResponse callback: fires on every attempt (including proxy-switching retries), letting you observe per-attempt status codes and attempt counts.

package main

import (
    "log"

    "github.com/cybergodev/httpc"
)

func main() {
    cfg := httpc.DefaultConfig()
    cfg.Connection.ProxyPool = []string{
        "http://proxy1:8080",
        "http://proxy2:8080",
        "http://proxy3:8080",
    }
    cfg.Connection.ProxyRotateOnStatus = []int{403}

    client, err := httpc.New(cfg)
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    result, err := client.Get("https://protected-site.example.com/data",
        httpc.WithOnResponse(func(resp httpc.ResponseMutator) error {
            log.Printf("attempt %d got %d", resp.Attempts(), resp.StatusCode())
            return nil
        }),
    )
    if err != nil {
        log.Fatal(err)
    }

    // Output: final proxy: http://proxy2:8080, total attempts: 2
    log.Printf("final proxy: %s, total attempts: %d", result.Meta.ProxyURL, result.Meta.Attempts)
}

Callbacks run per attempt, middleware runs per request

WithOnResponse fires inside the engine, on every attempt (including retries); the middleware chain wraps the whole retry cycle and runs once per logical request. Use callbacks for per-attempt observation and middleware for whole-request auditing.

Proxy and Retry Interaction

Proxy rotation and the retry engine are deeply integrated; two rules are worth memorizing.

1. The retry budget is raised automatically. When ProxyRotateOnStatus or ProxyRotatePerRequest is set and the pool holds more than one proxy:

text

effective MaxRetries = max(configured MaxRetries, len(ProxyPool) - 1) (capped at 10)

For example, 5 proxies with MaxRetries = 3: the budget is raised to 4 (= 5 - 1); the first request uses proxy1, and after a 403 it switches through proxy2…proxy5, trying each proxy exactly once.

2. Proxy connection failures are force-retried. While rotation is active, connection failures of the proxy itself (dial failure, TLS failure) bypass the regular retryability classification and go straight into the next retry on the next proxy — permanently dead proxies (wrong port, unreachable host) do not need to be pruned by hand; forced retries plus consecutive-failure circuit breaking eliminate them naturally.

For retry conditions, the backoff math, and the total timeout budget shared across retries, see Retry and Fault Tolerance.

Security Notes for Proxy Scenarios

The proxy features handle the following security details automatically — no manual configuration needed:

SSRF exemption: proxy host addresses (ProxyURL and every ProxyPool entry) are added to the SSRF exemption list automatically and are never blocked by private-IP checks — local proxies (e.g. 127.0.0.1:7890) work out of the box.
Deduplication: pool entries with the same host:port are merged automatically, avoiding rotation skew and double counting.
URL validation: every proxy URL goes through security validation (protocol whitelist http/https/socks5/socks5h, non-empty host, injection protection); an invalid value makes New() return an error instead of being silently ignored.

Relationship with TLS: an HTTPS request through an HTTP proxy's CONNECT tunnel is still end-to-end TLS — certificate validation, minimum TLS version, and certificate pinning all apply to the target site as usual, and the proxy only ever forwards ciphertext. If local DNS pollution is a concern, prefer socks5h (domain resolution delegated to the proxy); with DoH enabled, business domains resolve over the encrypted channel, but the proxy address itself does not go through DoH (proxies are configured explicitly by the developer and dialed directly). For the full SSRF picture, see SSRF Protection.

Common Issues

Problem	Cause	Solution
Proxy not taking effect	Both `ProxyURL` and `ProxyPool` set; `ProxyURL` wins	Clear `ProxyURL` and use only `ProxyPool`
Egress IP unchanged for consecutive requests to the same host	Connection reuse pinned the previous proxy tunnel	Enable `ProxyRotatePerRequest`
Proxies circuit-breaking frequently	`ProxyFailureThreshold` too low	Increase the threshold or `ProxyCooldown`
Status-code rotation not working	`Retry.MaxRetries = 0`, or the pool has only 1 proxy	Set `MaxRetries > 0`; keep at least 2 proxies in the pool
What happens when all proxies are broken	The whole pool failed consecutively	The engine falls back to the proxy closest to recovery instead of failing outright; check proxy availability and thresholds
System proxy not detected	Environment variables missing and platform settings off	Verify `HTTP_PROXY`/`HTTPS_PROXY` or the system proxy switch
Localhost proxy blocked by SSRF	The system proxy's exemption list is probed only at construction	Set `Connection.ProxyURL` explicitly (always exempt) or configure `SSRFExemptCIDRs`
Was the proxy circuit-broken after a 403 switch?	Mistakenly assuming status codes trigger circuit breaking	No — status-code rotation never breaks a proxy; only connection-level failures do
`Meta.ProxyURL` always empty	Only the system proxy is in effect — `EnableSystemProxy` is not recorded in this field	Use an explicit `ProxyURL` or `ProxyPool` when you need to observe the egress

For the complete field reference, see Configuration API — Proxy Pool.

Best Practices

Scenario	Recommended configuration
Fixed corporate egress	`ProxyURL` (http/https/socks5 as needed)
Follow user system settings	`EnableSystemProxy` + `NO_PROXY` allowing internal ranges
Multi-proxy load dispersion	`ProxyPool` + the default round-robin strategy
Scraping the same host	`ProxyPool` + `ProxyRotatePerRequest`
CF/WAF IP blocking	`ProxyPool` + `ProxyRotateOnStatus: []int{403}`
Mixed proxy quality	Increase `ProxyFailureThreshold` (tolerate flaps) and `ProxyCooldown`
Auditing the egress IP	Read `Result.Meta.ProxyURL`, combined with `WithOnResponse` for per-attempt observation
Managing proxy credentials	Keep them in the URL userinfo (auto-masked in logs); store the configuration file separately

The performance cost of rotation

Both ProxyRotatePerRequest and the retry path of status-code rotation close idle connections, trading connection reuse for egress rotation. For performance-sensitive scenarios that do not need rotation, keep the defaults (reuse connections); see Connection Pool & DNS for idle-connection management.

Next Steps

Connection Pool & DNS - How connection reuse interacts with proxy rotation, and DoH resolution
Retry and Fault Tolerance - Retry conditions, backoff algorithm, and proxy-pool interplay
Performance Optimization - The performance cost of proxy rotation and overall tuning
Configuration API - ConnectionConfig proxy field reference

Proxy & Proxy Pool ​

Proxy Mode Overview ​

Single Proxy Configuration ​

Authentication and Masking ​

System Proxy Detection and NO_PROXY ​

Platform Differences ​

Detection Order and Details ​

NO_PROXY Bypass Rules ​

Proxy Pool ​

Basic Usage ​

Configuration Fields ​

Selection Strategy ​

Passive Circuit Breaking ​

Status-Code Rotation ​

Per-Request Rotation ​

Deterministic Rotation ​

Finding Out Which Proxy a Request Used ​

Proxy and Retry Interaction ​

Security Notes for Proxy Scenarios ​

Common Issues ​

Best Practices ​

Next Steps ​