Rotating Proxies with Python Requests: A Copy-Paste Starter Kit

Introduction

If you're scraping, monitoring prices, or checking rank data at any real volume, a single IP will get rate-limited or blocked fast. Rotating proxies across requests is the standard fix, but most tutorials either hand-wave the retry logic or skip authentication entirely. This is a copy-paste starter kit: four increasingly useful patterns for rotating proxies with Python's requests library, from "just make it work" to "make it survive failures." Prerequisites pip install requests. You'll need a pool of proxy addresses. Any provider that gives you host:port (or user:pass@host:port for authenticated proxies) works with the patterns below - for these examples we're using the format <PROXY_HOST>:<PROXY_PORT>, which is what you'd swap in from a provider like Squid Proxies.

Pattern 1: Minimum Setup

The absolute minimum involves a single proxy handling both HTTP and HTTPS traffic simultaneously. By defining a proxy dictionary and passing it to requests.get, the library routes traffic through the chosen endpoint regardless of protocol.

import requests

proxy = " http://<PROXY_HOST>:<PROXY_PORT> "
proxies = {
    " http ": proxy,
    " https ": proxy,
}
response = requests.get(
    " https://httpbin.org/ip",
    proxies=proxies,
    timeout=10
)
print(response.json())

This approach satisfies basic needs-requests automatically handles both HTTP and HTTPS through the same proxy configuration.

Pattern 2: Round-Robin Pool Rotation

A single proxy isn't true rotation; it merely adds indirection. For genuine rotation, maintain a pool and cycle through it systematically. Using itertools.cycle creates an infinite iterator that cycles through the list repeatedly, ensuring each proxy gets used in turn.

import requests
from itertools import cycle

PROXIES = [
    " http://user:p***@proxy1.squidproxies.com:8000",
    " http://user:p***@proxy2.squidproxies.com:8000",
    " http://user:p***@proxy3.squidproxies.com:8000",
]

def get_proxy_pool(proxies):
    return cycle(proxies)

pool = get_proxy_pool(PROXIES)

for _ in range(5):
    proxy = next(pool)
    proxies = {
        " http ": proxy,
        " https ": proxy,
    }
    resp = requests.get(
        " https://httpbin.org/ip",
        proxies=proxies,
        timeout=10
    )
    print(proxy, "->", resp.json())

The cycle() function guarantees sequential rotation: proxy1 → proxy2 → proxy3 → proxy1 → proxy2 → … If unpredictable request timing is a concern, replace next(pool) with random.choice(PROXIES) for randomized selection.

Pattern 3: Retries with Exponential Backoff

Individual proxies can fail due to temporary blocking, timeouts, or rate limits. A robust solution combines rotation with retry logic that backs off exponentially between attempts, giving problematic proxies time to recover.

import time
import random
import requests

def make_request(url, proxy, timeout=10):
    proxies = {
        " http ": proxy,
        " https ": proxy,
    }
    try:
        resp = requests.get(url, proxies=proxies, timeout=timeout)
        resp.raise_for_status()
        return resp
    except requests.exceptions.RequestException as e:
        print(f"Request failed via {proxy}: {e}")
        return None

def fetch_with_rotation(url, proxies, max_retries=3):
    tried = []
    for attempt in range(max_retries):
        proxy = random.choice(proxies)
        tried.append(proxy)
        resp = make_request(url, proxy)
        if resp is not None:
            return resp
        time.sleep(2 ** attempt)  # 1s, 2s, 4s...
    raise RuntimeError(f"All {max_retries} attempts failed. Tried: {tried}")

response = fetch_with_rotation(
    " https://httpbin.org/ip",
    PROXIES
)
print(response.json())

Key design choices include raise_for_status() to convert HTTP error codes (403, 429, 503) into exceptions, preventing silent failures. Exponential backoff (2 ** attempt) gradually increases wait times between retries, reducing pressure on rate-limited endpoints. The tried list captures which proxies were attempted during failure, enabling faster diagnosis of consistently problematic sources.

Pattern 4: Reusable Session Class

When making numerous requests within a single script, encapsulating proxy management inside a class preserves connection pooling while allowing per-call rotation. This abstraction also serves as a foundation for adding additional features such as custom headers or domain-specific proxy assignments.

import random
import requests

class RotatingProxySession:
    def __init__(self, proxies):
        self.proxies = proxies
        self.session = requests.Session()

    def get(self, url, **kwargs):
        proxy = random.choice(self.proxies)
        self.session.proxies = {
            " http ": proxy,
            " https ": proxy,
        }
        return self.session.get(url, **kwargs)

    def post(self, url, **kwargs):
        proxy = random.choice(self.proxies)
        self.session.proxies = {
            " http ": proxy,
            " https ": proxy,
        }
        return self.session.post(url, **kwargs)

client = RotatingProxySession(PROXIES)

for i in range(5):
    resp = client.get(" https://httpbin.org/ip")
    print(f"Request {i}: {resp.json()}")

This pattern is particularly suited for production scraping jobs where the overhead of creating new connections on each request becomes significant. The class-based structure maintains consistent proxy rotation behavior across all calls while keeping the underlying requests.Session alive for efficient connection reuse.

Summary

Rotation resolves the common issue of a single IP being blocked or throttled; retries with exponential backoff address the reality that even healthy proxies may experience temporary failures. Combining both strategies provides resilience against both global rate limiting and individual proxy degradation. If you prefer asynchronous operations later, the same patterns translate naturally to aiohttp. The following link contains proxy pools formatted for use with https://www.squidproxies.com/.