How to Use Proxies with Python Requests

Python's requests library makes routing traffic through a proxy look like a one-line change, and for a quick test it is. In production the details start to matter: which key in the proxies dictionary maps to which traffic, how credentials with special characters get parsed, why a stray environment variable silently overrides your session settings, and what to do when the proxy itself times out or answers with a 407. This guide walks through every piece of a working setup, from the first proxied GET to a resilient session with retries, SOCKS5 support and an IP check you can trust.

Everything below uses the standard requests API plus urllib3's retry helper, which ships with it. If you are automating a real browser instead, our guide to using proxies with Selenium covers that side, and rotating proxies in Python goes deeper on pool management once the basics here are in place.

How Proxies Work in Python Requests

When you call requests.get(url, proxies=...), the library looks at the scheme of the target URL, picks the matching entry from your proxies dictionary, and opens its TCP connection to that proxy instead of the target host. What happens next depends on the target scheme.

• Plain HTTP targets: requests sends the full request line (including the absolute URL) to the proxy, and the proxy fetches the page on your behalf. The proxy can see and modify everything.
• HTTPS targets: requests first sends an HTTP CONNECT host:443 to the proxy, which opens a raw tunnel. TLS is then negotiated end to end between your script and the target, so the proxy only sees the hostname and the volume of encrypted traffic, not the content.
• SOCKS targets: with the optional SOCKS extra installed, requests speaks the SOCKS protocol to the proxy, which relays a TCP stream for any scheme.

This is why the most common confusion in proxy setups is the https key. The key names the scheme of the site you are visiting; the value names the scheme of the proxy itself. Most commercial HTTP proxies accept CONNECT over a plain HTTP connection, so the value for the https key usually still begins with http://.

Your First Proxied Request

The minimal working example routes both HTTP and HTTPS targets through the same proxy and prints the IP address the target saw:

import requests

proxy = "http://203.0.113.10:8080"
proxies = {
    "http": proxy,
    "https": proxy,
}

resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15)
print(resp.status_code, resp.json())

The address 203.0.113.10 is a documentation placeholder; swap in the host and port your provider gives you. If the printed origin IP is the proxy's address rather than your own, routing works. If you only set the http key, HTTPS requests will bypass the proxy entirely and go out from your real IP, which is a classic silent failure.

Routing per host

The proxies dictionary also accepts more specific keys. A key such as "https://example.com" applies only to that host, and "all://"-style catch-alls can be expressed with the all key. Specific keys win over generic ones, which lets you send one domain through a residential exit and everything else through a cheaper datacenter proxy without maintaining two clients.

Anatomy of a Proxy URL

Every proxy value is a URL with up to five parts: scheme, username, password, host and port. Understanding each part saves a lot of debugging.

Rotating residential gateways frequently use the username as a control channel. Webshare's documentation, for example, shows country and city targeting appended to the username (in the form username-us-city_los_angeles) against a single gateway host. Other providers use similar conventions for sticky session IDs. Check your provider's dashboard for the exact syntax rather than guessing; a malformed username usually produces a 407 rather than a helpful error.

Authenticated Proxies

There are two common authentication models, and requests handles both.

Username and password in the URL

Put credentials directly in the proxy URL. requests extracts them and sends a Proxy-Authorization header for you:

from urllib.parse import quote
import requests

user = "my_user"
password = quote("p@ss:word/with#chars", safe="")
proxy = f"http://{user}:{password}@proxy.example.com:8080"

proxies = {"http": proxy, "https": proxy}
resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15)

The quote(..., safe="") call matters. Characters like @, :, / and # have structural meaning inside a URL; left unencoded, they make the parser split the string in the wrong place and you get an authentication failure that looks like a bad password.

IP allowlisting

Many providers let you register your server's public IP in their dashboard instead of using credentials. In that case the proxy URL is just http://host:port. Allowlisting is convenient on fixed servers, but it breaks the moment your egress IP changes, for example on a laptop, a CI runner or an autoscaling container fleet. Credentials travel with the code; allowlists travel with the machine.

Using SOCKS5 Proxies

requests does not speak SOCKS out of the box. Install the optional extra, which pulls in PySocks:

pip install "requests[socks]"

Then use a SOCKS scheme in the proxy URL:

proxy = "socks5h://user:pass@proxy.example.com:1080"
proxies = {"http": proxy, "https": proxy}
resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15)

The difference between socks5:// and socks5h:// is where DNS happens. With socks5://, your machine resolves the hostname and sends the proxy an IP address, which means your local resolver (and possibly your ISP) sees every domain you visit. With socks5h://, the hostname is passed to the proxy and resolved there. For privacy and for geo-accurate DNS answers, socks5h is usually what you want. Our SOCKS5 vs HTTP proxies comparison explains when SOCKS is worth it at all; for most web scraping, an HTTP proxy with CONNECT does the same job with fewer moving parts.

Sessions, Environment Variables and Precedence

For anything beyond a one-off request, use a requests.Session. It reuses TCP connections (including the tunnel to the proxy), keeps cookies, and lets you set defaults once:

import os
import requests

session = requests.Session()
session.proxies.update({
    "http": os.environ["PROXY_URL"],
    "https": os.environ["PROXY_URL"],
})
session.headers.update({"User-Agent": "my-crawler/1.0 (+contact@example.com)"})

resp = session.get("https://httpbin.org/ip", timeout=15)

The environment-variable trap

requests also honors the standard HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and NO_PROXY environment variables, because Session.trust_env is True by default. That is handy for configuring tools without code changes, but it creates a precedence surprise: when requests merges settings, a proxy passed directly to the request call wins, environment variables come next, and values set on session.proxies come after them. So a forgotten HTTPS_PROXY in a Docker image or CI config can quietly override the proxy you set on the session.

Two reliable fixes:

• Pass proxies=... on each request call (or wrap the session in a helper that does), which always wins.
• Set session.trust_env = False when you want the session's configuration to be the only source of truth. Note that this also disables reading .netrc credentials and environment CA bundle settings.

NO_PROXY is the flip side: hosts listed there (comma-separated, such as localhost,127.0.0.1,.internal.example) bypass the proxy. It is useful for keeping internal API calls off your paid bandwidth.

Timeouts, Errors and Retries

requests has no default timeout. Without one, a dead proxy can hang a worker indefinitely. Always pass timeout, ideally as a tuple of (connect, read) seconds, for example timeout=(5, 30): a short connect timeout fails fast on unreachable proxies, while the read timeout allows slow pages to finish.

Exceptions you will actually see

Automatic retries with backoff

Mount an HTTPAdapter configured with urllib3's Retry to handle transient failures without hand-written loops:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=3,
    backoff_factor=0.5,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
)
adapter = HTTPAdapter(max_retries=retry)

session = requests.Session()
session.mount("http://", adapter)
session.mount("https://", adapter)

This retries the same proxy, which is right for brief network blips. For blocks (403, CAPTCHA pages, repeated 429s), retrying through the same exit rarely helps; you want a fresh IP, which is where rotation comes in.

Rotating Proxies with Requests

There are two ways to get a new IP per request or per task.

1. Gateway rotation. Rotating residential products typically give you one hostname that assigns a different exit IP per connection, or keeps the same exit for a sticky window when you add a session ID to the username. Your code stays simple: one proxy URL, and the provider handles the pool.
2. Client-side rotation. With a list of static proxies (datacenter or ISP), you choose one per request yourself:

import random
import requests

POOL = [
    "http://user:pass@198.51.100.11:8080",
    "http://user:pass@198.51.100.12:8080",
    "http://user:pass@198.51.100.13:8080",
]

def fetch(url, attempts=3):
    last_exc = None
    for _ in range(attempts):
        proxy = random.choice(POOL)
        try:
            r = requests.get(url, proxies={"http": proxy, "https": proxy},
                             timeout=(5, 30))
            if r.status_code in (403, 429):
                continue
            return r
        except requests.exceptions.RequestException as exc:
            last_exc = exc
    raise RuntimeError(f"all attempts failed: {last_exc}")

Real pools need more than random.choice: health tracking, cooldowns for IPs that just got blocked, and per-domain concurrency limits. Our proxy rotation guide builds that out. Remember that rotation fights the wrong problem on login flows; if a site expects one user to keep one IP for a session, use sticky sessions instead, as covered in sticky sessions explained.

Verifying Your Proxy Is Actually Working

A setup that returns 200 is not necessarily a setup that uses the proxy. Build a small check into your startup code:

import requests

def exit_ip(proxies=None):
    r = requests.get("https://api.ipify.org?format=json",
                     proxies=proxies, timeout=(5, 15))
    r.raise_for_status()
    return r.json()["ip"]

direct = exit_ip()
proxied = exit_ip({"http": PROXY, "https": PROXY})
assert direct != proxied, "proxy not applied"
print("direct:", direct, "proxied:", proxied)

Compare the direct and proxied IPs. If they match, something is bypassing the proxy (a missing https key, NO_PROXY, or trust_env settings). For location-sensitive jobs, also check the proxied IP against a geolocation lookup and confirm the country matches what you paid for, especially on rotating gateways where a typo in the targeting username can silently fall back to a random country.

Requests vs httpx for Proxied Workloads

requests is synchronous: each request blocks a thread until it completes. That is fine for scripts and modest crawlers, but high-concurrency scraping through slow residential exits benefits from async I/O. The httpx library offers a very similar API with both sync and async clients. In current httpx versions, a single proxy is passed with the proxy= argument (older releases used a proxies= mapping), and per-scheme routing is done with mounts:

import httpx

with httpx.Client(proxy="http://user:pass@proxy.example.com:8080",
                  timeout=15) as client:
    print(client.get("https://httpbin.org/ip").json())

If your bottleneck is waiting on network round-trips, moving to an async client usually buys more throughput than adding more proxies.

Choosing a Proxy Type for Requests-Based Scrapers

The library does not care what kind of IP sits behind the proxy URL, but the target site does. Match the IP type to how defended the target is:

• Datacenter proxies are fast and cheap, and work well on APIs, public datasets and sites with light bot protection. Their ASN reveals a hosting provider, so heavily defended sites block them more readily.
• ISP (static residential) proxies carry consumer ISP registration with server-grade stability, a good fit for logged-in sessions and long-lived identities.
• Rotating residential proxies draw from large pools of household IPs, billed by bandwidth, and handle tougher anti-bot systems at a higher cost per gigabyte.

Our residential vs ISP vs datacenter breakdown covers the trade-offs in depth. Two providers that pair well with a requests-based stack are below.

Webshare: self-service datacenter and residential

Webshare suits developers who want to start immediately: a genuinely free tier for testing, a plan builder that lets you buy exact proxy counts, and HTTP and SOCKS5 endpoints that drop straight into a proxies dict.

IPRoyal: pay-as-you-go with SOCKS5 everywhere

IPRoyal fits intermittent scrapers. Residential traffic you buy does not expire, and SOCKS5 is supported across its product range, which is convenient if you standardize on socks5h://.

Common Mistakes to Avoid

• Setting only the http key. HTTPS traffic then skips the proxy and leaks your real IP.
• Using https:// as the proxy scheme by habit. Unless your provider explicitly offers TLS to the proxy, the value should start with http:// even for HTTPS targets.
• No timeout. One dead proxy can freeze a worker pool.
• Disabling certificate verification to make errors go away. verify=False exposes you to interception and hides real configuration bugs. Fix the scheme or CA bundle instead.
• Retrying blocks on the same IP. Retries are for transient errors; blocks need a new exit or a slower pace.
• Ignoring site rules. A proxy changes your IP, not your obligations. Respect robots directives where relevant, terms of service and rate limits, and avoid collecting personal data you do not need.

Conclusion

Getting proxies working in Python Requests comes down to a handful of habits: a complete proxies dictionary, URL-encoded credentials, explicit timeouts, deliberate handling of environment variables, retries for transient errors and fresh IPs for blocks, plus an exit-IP check before every serious run. With those in place, swapping between datacenter, ISP and residential providers becomes a one-line configuration change rather than a debugging session. When your crawler outgrows a single proxy, move on to structured rotation and pick the IP type that matches how hard your targets push back.