HTTP and web APIs

A web API is a service that other programs talk to over HTTP: you send a request to a URL and get back data, usually JSON. Weather data, payments, maps, package indexes and chat services all work this way. This unit teaches the client side: how HTTP requests and responses are built, how to send them with the standard library's urllib and with the popular requests package, and how to behave well when the network or the server misbehaves.

Every example runs offline. Each program starts a tiny web server of its own on your computer, talks to it, and shuts it down. In real code you replace that server's address with the API's URL; nothing else changes.

You will be able to:

• Describe an HTTP request and response: method, URL, headers, status code and body.
• Take URLs apart and build them safely with urllib.parse.
• Send GET and POST requests with urllib.request and with requests, always with a timeout, and handle the errors each one raises.
• Use a requests.Session with default headers and a token read from an environment variable.
• Retry failed requests with exponential backoff, and respect rate limits and terms of service.

How HTTP works

HTTP is a request and response protocol. The client sends a request: a method, a URL, some headers and sometimes a body. The server sends back a response: a status code, headers and usually a body.

The method says what you want to do with the resource the URL names:

Safe means the request does not change anything on the server. Idempotent means sending it twice has the same effect as sending it once, which matters when you retry: repeating a GET is harmless, repeating a POST may create two orders.

The status code tells you how it went. The first digit gives the class: 1xx informational, 2xx success, 3xx redirection, 4xx a problem with your request, 5xx a problem on the server. The http.HTTPStatus enum knows them all (the is_* properties arrived in 3.12):

Other codes you will meet often are 204 No Content, 301 and 302 (redirects), 304 Not Modified, 400 Bad Request, 401 Unauthorized (no or bad credentials), 403 Forbidden (valid credentials, not allowed), 422 Unprocessable Content (well formed but invalid data), 500 Internal Server Error, 502 Bad Gateway and 504 Gateway Timeout.

Headers are name and value pairs; their names are case insensitive. The important ones for API work:

A JSON body is UTF-8 text with Content-Type: application/json. You turn Python objects into it with json.dumps() and back with json.loads(), as in Files and data formats.

URLs with urllib.parse

A URL has several parts: scheme://host:port/path?query#fragment. urllib.parse splits URLs, builds query strings with correct escaping, and resolves relative links. It does no networking, so it is safe to experiment with.

What to notice:

• parse_qs() returns a list for every key, because a key may repeat. urlencode() escapes spaces, &, non ASCII letters and everything else that would break the URL; with doseq=True a list becomes repeated keys.
• quote() escapes a path and leaves / alone by default. Pass safe="" when you escape a single segment, so that a / inside it is escaped as well.
• urljoin() resolves a link the way a browser does. Watch the trailing slash: without it, guide is treated as a file and replaced.

Pitfall. Never build a query string by gluing user input into the URL (f"...?q={text}"). A space, & or # in the text changes the request. Use urlencode(), or the params= argument of requests.

Requests with urllib.request

urllib.request is the standard library's HTTP client. It needs no installation, which makes it handy for small scripts and tools. Every example below begins with the same kind of local server: a ThreadingHTTPServer on 127.0.0.1 with port 0, so the operating system picks a free port, running in a background thread. Its handler class answers requests with JSON. You do not need to understand the server part to follow the client code; it ends at the line that sets BASE.

What to notice:

• urlopen() returns a response object to use in a with block. .status is the status code (the old getcode() method is deprecated), .headers the headers, and .read() returns the body as bytes; json.load(response) reads and parses it in one step.
• A Request object lets you set the method, headers and body. The body must be bytes, so encode the JSON text.
• Every call has a timeout. Without one, a server that never answers makes your program wait forever.
• Error statuses do not return normally: urlopen() raises HTTPError, which carries the code, the headers and the body.

Pitfall. HTTPError is a subclass of URLError, so the order of the except clauses matters: catch HTTPError first, or the URLError clause swallows it. A connection that opens but then stalls raises TimeoutError, which is not a URLError at all; catch OSError if you want one clause for every network failure.

The requests library

requests is a third party package with a friendlier interface and features such as sessions, automatic JSON handling and pluggable retries. Install it in your virtual environment:

The example server below adds three things: a paged list of books, validation on POST, and a /me endpoint that requires a token. The token is read from an environment variable, which is where credentials belong: never in the source code. Set it in your shell before running a real script.

What to notice:

• params= builds and escapes the query string; json= serialises the body and sets Content-Type: application/json. Use data= instead for HTML form encoding.
• Unlike urlopen(), requests does not raise for error statuses. Check response.status_code or response.ok, or call response.raise_for_status(), which raises requests.HTTPError for 4xx and 5xx.
• response.json() parses the body into dicts and lists; response.text is the body decoded to str and response.content the raw bytes. response.headers ignores case, so "content-type" works.
• A Session reuses the network connection between requests and applies its headers to every call. Use one whenever you make several requests to the same API.
• timeout=5 limits each wait for the server; a pair such as timeout=(3, 30) sets the connection and read limits separately. Without timeout, requests waits indefinitely.

The requests exceptions all derive from requests.RequestException, itself an OSError:

Retries, backoff and rate limits

Networks fail in passing: a connection drops, a server restarts, a gateway times out. Retrying such transient failures is reasonable. Retrying a 400 Bad Request is not, since the same request will fail the same way. A sound policy:

• Retry connection errors, timeouts, and the statuses 429, 500, 502, 503 and 504.
• Wait longer after each failure (exponential backoff: 0.5 s, 1 s, 2 s, ...), with some randomness (jitter) so that many clients do not retry in lockstep.
• If the server sends Retry-After, wait that long instead.
• Give up after a few attempts, and retry non idempotent requests such as POST only when the API says it is safe.

The program below implements that policy by hand, then lets the Retry class from urllib3, the library underneath requests, do the same job. Its server fails twice per path and then succeeds. The delays are tiny so the example runs fast.

The second wait was 0 because the server's Retry-After: 0 overrode the backoff. A real Retry-After can also be an HTTP date, which this helper treats as absent. urllib3's Retry retries only idempotent methods by default, honours Retry-After for 413, 429 and 503, and otherwise waits backoff_factor × 2, × 4, ... seconds between later attempts (the first retry follows at once).

Rate limits. Most public APIs cap how many requests you may send per second, hour or day, and answer 429 Too Many Requests when you exceed it. Read the API's documentation for its limits, keep below them (for example, wait a fixed interval between calls, measured with time.monotonic()), cache answers you will need again, and back off when you get a 429.

Terms of service. An API's terms say what you may do with it and its data; follow them, and use the official API rather than scraping web pages when one exists. Do not scrape search engines or sites whose terms forbid automated access, check robots.txt (the standard library's urllib.robotparser reads it), and identify your program with a descriptive User-Agent that includes a way to contact you.

Tip. For asynchronous code (Concurrency), the third party httpx package offers an API close to requests in both a synchronous form and an async one (httpx.AsyncClient), so you can send many requests concurrently.

Check yourself

1. What is the difference between a safe and an idempotent method? Why does it matter for retries?
2. What do 401, 403, 404 and 429 tell you?
3. Why should you build query strings with urlencode() or params= instead of f-strings?
4. Why must except HTTPError come before except URLError?
5. urlopen() raises an exception for a 404; requests.get() does not. How do you detect the error with requests?
6. What happens if you leave out timeout= in requests.get()?
7. Which failures should you retry, and why add jitter to the backoff delay?
8. Where should an API token come from, and why?

Suggested answers

1. A safe method changes nothing on the server; an idempotent one has the same effect whether sent once or several times. Retrying an idempotent request cannot do extra harm; retrying a POST might create duplicates.
2. 401: credentials missing or wrong. 403: you are known but not allowed. 404: no resource at that URL. 429: too many requests; slow down, and check Retry-After.
3. Because spaces, &, # and non ASCII characters must be escaped, or they change the meaning of the URL. The library functions escape every value.
4. HTTPError is a subclass of URLError, so an earlier except URLError clause would catch HTTP errors too, and the HTTPError clause would never run.
5. Check response.status_code or response.ok, or call response.raise_for_status(), which raises requests.HTTPError for 4xx and 5xx.
6. The request can wait forever if the server accepts the connection but never answers, freezing your program.
7. Connection errors, timeouts, 429 and 5xx statuses such as 502, 503 and 504, which are usually temporary; not other 4xx errors. Jitter spreads retries from many clients over time, so they do not hit a recovering server all at once.
8. From an environment variable (or a secrets manager), never from the source code, so it is not leaked through version control and can be changed without editing the program.

Exercises

These exercises are optional practice. Pick the ones that help you.

Exercises 2 to 5 run against a local server, as the examples do. Each solution starts its own. You are not expected to write these servers. Copy the server part from the top of the matching solution (everything down to the line that sets BASE), then write the client function yourself.

1. Build a search URL. Write with_query(url, **params) that adds the parameters to a URL's query string, keeps any query the URL already has, skips parameters whose value is None, and repeats a key for list values. with_query("https://api.example.com/v1/search?lang=en", q="green tea", tag=["a", "b"], page=None) returns "https://api.example.com/v1/search?lang=en&q=green+tea&tag=a&tag=b".
2. Fetch a title with urllib. Write get_title(base, book_id) that requests /books/<id> with urllib.request and returns the book's title, or None when the server answers 404. Any other error should propagate. Always pass a timeout.
3. Create a book with requests. Write create_book(session, base, title) that posts {"title": ...} as JSON to /books and returns the new book's id. Strip the title and raise ValueError for a blank one without sending anything. Raise for any error status. On a fresh server, create_book(session, base, " Night Garden ") returns 1 and the server receives {"title": "Night Garden"}.
4. Collect every page. A server returns pages like {"items": [...], "next": "/books?page=2"}, with next set to None on the last page. Write fetch_all(session, base, first_path) that follows next links and returns all items in order. With five books and two per page it makes three requests.
5. Keep a polite pace. Write a class PacedSession that wraps a requests.Session and guarantees at least min_interval seconds between the start of consecutive requests, measured with time.monotonic(). Record the start times so that you can check the gaps, and use a server that counts the requests it receives.
6. Optional: a real API. The Python Package Index publishes JSON about each project at https://pypi.org/pypi/<name>/json. Write latest_version(name) that returns data["info"]["version"]. Use a timeout, and raise for error statuses. This one needs network access.

Solutions

1. Build a search URL.

Working with a list of pairs, rather than a dict, keeps the order and allows repeated keys.

2. Fetch a title with urllib.

Only the 404 is turned into None; a 500 or a network failure still raises, so the caller cannot mistake an outage for a missing book.

3. Create a book with requests.

Validating before sending saves a round trip and gives a clearer error than the server's.

4. Collect every page.

Following the server's next link, instead of counting pages yourself, keeps working if the API changes its page size or its URL scheme.

5. Keep a polite pace.

The while loop rechecks the clock after sleeping, so the gap is never shorter than asked even if a sleep ends a little early. time.monotonic() is the right clock here: it cannot jump if the system time is corrected. The server only counts requests; timing them on the server side would add network jitter to the measurement.

6. Optional: a real API.

The result changes whenever the project publishes a release. A name that does not exist on PyPI gives a 404, which raise_for_status() turns into requests.HTTPError.