Web

Sending HTTP Requests in Python with urllib3

Web · Published 2026-10-24

To send an HTTP request with urllib3, create one urllib3.PoolManager() and call http.request("GET", url); the returned response exposes .status, .headers, the raw bytes in .data and, since urllib3 2.0, a .json() helper. Everything else (query parameters, JSON bodies, timeouts, retries) is a keyword argument on that same call.

urllib3 is the connection-pooling library underneath requests and pip. Using it directly gives you fewer dependencies and fine-grained control over pools, retries and timeouts. The examples below were run against urllib3 2.8.0 on Python 3.13, using httpbin.org as a test server. Current 2.x releases require Python 3.10 or newer; install with pip install urllib3.

PoolManager: the object you reuse

A PoolManager keeps a pool of open connections per host and reuses them across requests (HTTP keep-alive). Create one per application, not one per request, and share it. It is thread-safe.

import urllib3

http = urllib3.PoolManager()

resp = http.request("GET", "https://httpbin.org/get",
                    fields={"q": "loop branch", "page": "2"})

print(resp.status)
print(resp.headers["Content-Type"])
data = resp.json()
print(data["args"])
print(data["url"])

200
application/json
{'page': '2', 'q': 'loop branch'}
https://httpbin.org/get?q=loop+branch&page=2

For one-off scripts, urllib3 2.x also offers a module-level urllib3.request("GET", url) that uses a shared global pool manager. In libraries and services, prefer an explicit PoolManager so you control its configuration.

The response object

resp.data is always bytes. Decode text explicitly, e.g. resp.data.decode("utf-8"). Unlike requests, urllib3 does not guess the charset for you.

Query parameters with fields

For GET, HEAD and DELETE, the fields dictionary is URL-encoded into the query string, as the output above shows. For POST and PUT, fields is sent as a form body instead (multipart by default). If you need query parameters on a POST, encode them into the URL yourself:

from urllib.parse import urlencode

url = "https://httpbin.org/post?" + urlencode({"dry_run": "1"})
resp = http.request("POST", url, json={"id": 7})

POST requests: JSON, forms, files and raw bytes

urllib3 2.0 added the json= argument. It serializes the object and sets Content-Type: application/json unless you set one yourself.

resp = http.request(
    "POST",
    "https://httpbin.org/post",
    json={"name": "Ada", "tags": ["py", "web"]},
    headers={"User-Agent": "loop-branch-demo/1.0",
             "Authorization": "Bearer TOKEN"},
)
body = resp.json()
print(resp.status)
print(body["json"])
print(body["headers"]["Content-Type"])
print(body["headers"]["User-Agent"])

200
{'name': 'Ada', 'tags': ['py', 'web']}
application/json
loop-branch-demo/1.0

On urllib3 1.26 there is no json=; encode manually with body=json.dumps(obj).encode() and set the Content-Type header. That form still works on 2.x.

Form fields, file uploads and raw bodies:

# multipart/form-data form fields
resp = http.request("POST", "https://httpbin.org/post",
                    fields={"field": "value", "n": "1"})
print(resp.json()["form"])

# file upload: (filename, content, mime type)
resp = http.request("POST", "https://httpbin.org/post",
                    fields={"report": ("report.txt", b"hello\n", "text/plain")})
print(resp.json()["files"])

# raw bytes with your own content type
resp = http.request("POST", "https://httpbin.org/post",
                    body=b"raw bytes",
                    headers={"Content-Type": "application/octet-stream"})
print(resp.json()["data"])

{'field': 'value', 'n': '1'}
{'report': 'hello\n'}
raw bytes

To send application/x-www-form-urlencoded instead of multipart, pass encode_multipart=False together with fields.

Headers

Pass headers= per request, or set defaults on the pool manager with PoolManager(headers={...}). Note that per-request headers replace the pool defaults rather than merging with them. The helper urllib3.util.make_headers builds common headers:

from urllib3.util import make_headers

print(make_headers(basic_auth="user:pass", accept_encoding=True,
                   user_agent="demo"))

{'accept-encoding': 'gzip,deflate', 'user-agent': 'demo', 'authorization': 'Basic dXNlcjpwYXNz'}

With accept_encoding=True, urllib3 advertises only the codecs it can decode: br and zstd are added to the list when the optional Brotli and zstandard packages are installed (pip install "urllib3[brotli,zstd]").

Timeouts

By default urllib3 waits indefinitely for a slow server. Always set a timeout. A float applies to both connect and read; a Timeout object separates them.

from urllib3.util import Timeout

http = urllib3.PoolManager(timeout=Timeout(connect=2.0, read=5.0))

try:
    http.request("GET", "https://httpbin.org/delay/3",
                 timeout=Timeout(connect=2.0, read=1.0), retries=False)
except urllib3.exceptions.ReadTimeoutError as e:
    print("timeout:", type(e).__name__)

timeout: ReadTimeoutError

The read timeout applies to each socket read, not to the whole download, so a server that trickles bytes can exceed it in wall-clock time.

Retries with Retry

By default urllib3 retries up to three times on connection errors and follows redirects, but it does not retry on error status codes. Configure that with urllib3.util.Retry:

from urllib3.util import Retry

retry = Retry(
    total=3,
    backoff_factor=0.5,                   # sleeps 0.5s, 1s, 2s ... between tries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods={"GET", "HEAD"},      # never blindly retry POST
)
http = urllib3.PoolManager(retries=retry)

try:
    http.request("GET", "https://httpbin.org/status/503",
                 retries=Retry(total=2, backoff_factor=0.1,
                               status_forcelist=[503]))
except urllib3.exceptions.MaxRetryError as e:
    print("gave up:", e.reason)

r = http.request("GET", "https://httpbin.org/status/503",
                 retries=Retry(total=2, backoff_factor=0.1,
                               status_forcelist=[503], raise_on_status=False))
print("status after retries:", r.status)

gave up: too many 503 error responses
status after retries: 503

With raise_on_status=False you get the last response back instead of an exception. Retry honours Retry-After headers on 429/503 by default. A plain 404 is never retried and never raises; check resp.status yourself. Pass retries=False to disable retries and redirects entirely and see the raw exception.

Version note: the old method_whitelist argument was renamed allowed_methods in 1.26 and removed in 2.0.

Redirects

Redirects are followed automatically and count against the retry budget (Retry(redirect=n)). Disable them per request to inspect the Location header:

r = http.request("GET", "https://httpbin.org/redirect/2")
print(r.status, len(r.retries.history))

r = http.request("GET", "https://httpbin.org/redirect/2", redirect=False)
print(r.status, r.headers["Location"])

200 2
302 /relative-redirect/1

On cross-host redirects urllib3 strips Authorization, Cookie and Proxy-Authorization headers (configurable via Retry(remove_headers_on_redirect=...)).

TLS certificate verification

urllib3 2.x verifies HTTPS certificates by default using the system trust store via the ssl module, and requires TLS 1.2 or newer and OpenSSL 1.1.1+. To use a specific CA bundle, such as the one from certifi or a corporate CA:

import certifi

http = urllib3.PoolManager(cert_reqs="CERT_REQUIRED", ca_certs=certifi.where())

A failed handshake raises urllib3.exceptions.SSLError (wrapped in MaxRetryError if retries are enabled). Setting cert_reqs="CERT_NONE" disables verification and triggers an InsecureRequestWarning; do not use it outside throwaway local testing. Client certificates are passed with cert_file= and key_file=.

Streaming large responses

By default the whole body is read into memory. For large downloads, set preload_content=False and iterate with stream(), then return the connection to the pool:

r = http.request("GET", "https://httpbin.org/bytes/102400",
                 preload_content=False)
total = 0
with open("download.bin", "wb") as f:
    for chunk in r.stream(32 * 1024):
        f.write(chunk)
        total += len(chunk)
r.release_conn()
print("bytes written:", total)

bytes written: 102400

The response is also a file-like object, so shutil.copyfileobj(r, f) works. Forgetting release_conn() (or not draining the body) leaks a connection from the pool.

Exceptions

All urllib3 errors inherit from urllib3.exceptions.HTTPError (not to be confused with urllib.error.HTTPError). The ones you will meet most:

try:
    http.request("GET", "https://no-such-host.invalid/", retries=1)
except urllib3.exceptions.MaxRetryError as e:
    print("MaxRetryError ->", type(e.reason).__name__)

MaxRetryError -> NameResolutionError

What changed in urllib3 v2

• json= argument on request() and HTTPResponse.json().
• Module-level urllib3.request().
• TLS 1.2 minimum, OpenSSL 1.1.1+ required; the DEFAULT_CIPHERS list and old Python versions were dropped.
• method_whitelist and other deprecated Retry names removed; HTTPResponse.getheaders() deprecated in favour of resp.headers.
• The urllib3.contrib.pyopenssl and SecureTransport shims are deprecated or gone.

The full list is in the v2 migration guide. If an older dependency pins urllib3<2, code using json= or resp.json() will fail with a TypeError or AttributeError.

urllib3 vs requests vs http.client

Pick urllib3 when you want control and minimal dependencies, requests when developer convenience matters more, and http.client only when you cannot install anything.

Common pitfalls

• Expecting exceptions on 4xx/5xx. urllib3 returns error responses normally; check resp.status.
• No timeout. The default can hang forever; set one on the pool manager.
• Using fields on POST for query strings. It becomes a form body; put query parameters in the URL.
• Treating resp.data as str. It is bytes; decode it.
• A new PoolManager per request. This throws away connection reuse.

Summary

Create one PoolManager, call request(method, url, ...), and read status, headers, data or json(). Use fields for query strings on GET, json= for JSON bodies (v2), Timeout for bounded waits, Retry for back-off on transient failures, and preload_content=False with stream() for large downloads. Catch MaxRetryError and inspect e.reason for the underlying cause.

Read next

More in Web

Web

Scrapy vs Selenium

Scrapy vs Selenium compared: architecture, JavaScript rendering, speed, scaling and pipelines, with working code for each and when to use scrapy-playwright.

Web

What is a CRUD application?

CRUD means Create, Read, Update, Delete. See how it maps to SQL and REST HTTP methods, with a runnable Python example, validation, soft deletes and idempotency.

Python

'NoneType' object is not subscriptable

Fix 'NoneType' object is not subscriptable: common causes (list.sort(), dict.get, re.match, JSON null), reading the traceback, and catching it with mypy.