Blog / Data engineering

Handle Web Data API Errors in Python: 4xx, 5xx and Timeouts

22 Sept 2026Perceptlayer7 min read

Sort every failed response into one of three actions. Retry 429, 503 and 502 with a wait. Fix the request on 401, 403, 404 and 422. Stop the job on 402. Timeouts are the odd case: they carry no status code, and the call may have succeeded. This guide gives you the mapping and a Python function with the requests library that applies it.

Which API errors should you retry, fix or stop on?

Going by the code names, retry the ones about the moment (rate_limited, temporarily_unavailable, source_error). Fix the ones about your request (invalid_api_key, restricted_endpoint, not_found, invalid_parameters). Stop on the one about your account (insufficient_credits). The table adds the action we recommend to each code, and every row costs 0 credits. For what each code covers, see Errors in the docs.

The 502 row has no documented Retry-After, so the code below falls back to exponential backoff with jitter. Any other 5xx isn't on the documented list. Treating it as retryable with a hard cap is our policy, not a documented behaviour, and the cap is what keeps that safe.

How do you call the endpoint and read the response?

Send the key in the X-API-Key header and the parameters in the query string. The examples use the Google Search API search endpoint, which costs 1 credit per successful call. The same pattern works for any of the 1,659 endpoints across 248 websites, because every error uses the same body shape.

curl "https://api.perceptlayer.com/v1/google-search/search?query=best+restaurants+in+Berlin&hl=en&safe=off" \
  -H "X-API-Key: $API_KEY"

These are the parameters that matter for error handling. query is the only required one. Check your values against the documented limits before you send them. The full list is on the Search endpoint page.

A successful call returns JSON with named fields. This is an excerpt of a recorded response for the request above:

{
  "search_information": {
    "query_displayed": "best restaurants in Berlin",
    "organic_results_count": 6
  },
  "organic_results": [
    {
      "position": 1,
      "title": "The Best Restaurants in Berlin - Food Guide & Map",
      …
    }
  ],
  …
}

To see an error on purpose, send a wrong key with curl -i. Read the status and the error.code, and check the headers: errors carry X-Credits-Remaining, and failed calls return X-Credits-Used: 0. The -i flag prints the headers, which is where Retry-After and the RateLimit-* values also show up.

How do you write the retry loop in Python?

Write one function that every call goes through. It raises a custom exception for errors you must fix or stop on, retries the rest a fixed number of times, and sleeps for Retry-After when the header is there. It sets a timeout on every call, because requests waits forever without one (Requests documentation).

import logging
import os
import random
import time

import requests

BASE_URL = "https://api.perceptlayer.com"
RETRY_CODES = {"rate_limited", "temporarily_unavailable", "source_error"}

log = logging.getLogger("webdata")


class ApiError(Exception):
    """A response you must fix or stop on. Retrying won't help."""

    def __init__(self, status, code, message, request_id):
        super().__init__(f"{status} {code}: {message} (request_id={request_id})")
        self.status = status
        self.code = code
        self.request_id = request_id


class RetriesExhausted(Exception):
    pass


def parse_error(resp):
    try:
        err = resp.json()["error"]
        return err["code"], err["message"], err["request_id"]
    except (ValueError, KeyError, TypeError):
        return "unknown", "no error body", None


def wait_seconds(retry_after, attempt):
    try:
        return min(float(retry_after), 60)
    except (TypeError, ValueError):
        return min(2 ** attempt, 30) + random.random()


def call(path, params, attempts=4, timeout=(5, 30)):
    headers = {"X-API-Key": os.environ["API_KEY"]}
    for attempt in range(1, attempts + 1):
        retry_after = None
        try:
            resp = requests.get(
                BASE_URL + path, params=params, headers=headers, timeout=timeout
            )
        except requests.exceptions.ReadTimeout:
            # The request was sent. It may have succeeded and been billed.
            reason = "read timeout"
        except requests.exceptions.ConnectionError:
            # Includes connect timeouts: the request never reached the API.
            reason = "connection failed"
        else:
            if resp.ok:
                if resp.headers.get("RateLimit-Remaining") == "0":
                    time.sleep(float(resp.headers.get("RateLimit-Reset", 1)))
                return resp.json()
            code, message, request_id = parse_error(resp)
            if code not in RETRY_CODES and resp.status_code < 500:
                raise ApiError(resp.status_code, code, message, request_id)
            reason = f"{resp.status_code} {code} (request_id={request_id})"
            retry_after = resp.headers.get("Retry-After")

        log.warning("attempt %d of %d failed: %s", attempt, attempts, reason)
        if attempt < attempts:
            time.sleep(wait_seconds(retry_after, attempt))
    raise RetriesExhausted(f"gave up after {attempts} attempts: {reason}")

The function never logs a response body, only the status, code and request_id. Keep the request_id: it's what you quote if you ask for help with a specific failure.

Here's how a caller uses it:

try:
    data = call(
        "/v1/google-search/search",
        {"query": "best restaurants in Berlin", "hl": "en", "safe": "off"},
    )
    print(data["search_information"]["organic_results_count"])
except ApiError as err:
    if err.code == "insufficient_credits":
        raise SystemExit("Out of credits. Buy a pack, then rerun.")
    print(f"Fix the request: {err}")
except RetriesExhausted as err:
    print(f"Still failing, try again later: {err}")

Why not just use urllib3's Retry?

You can. requests sits on urllib3, and its Retry class mounts on a session with status_forcelist and respect_retry_after_header. It can't read error.code from the body or log the request_id. If you only need retries, use Retry. For the mapping above, the explicit loop is a few dozen lines.

How do you handle timeouts without paying twice?

A connect timeout is safe to retry because the request never reached the API. A read timeout isn't: the API may have finished the call and billed it, and your client just never saw the reply. Requests have no idempotency key, so the API can't tell a retry from a new call. Set both timeouts, retry with a cap, and accept that a retry after a read timeout can cost one extra credit.

The tuple timeout=(5, 30) in the code above means 5 seconds to connect and 30 to read. Pick the read value from what you see in your own traffic, then add headroom. Too tight and you turn slow successes into duplicate charges. Too loose and a stuck call holds a worker.

Two habits keep the extra credits small. Cap attempts: four attempts means at most three retries. And log read timeouts separately, because a rising count tells you the timeout is too tight. If you need to reconcile, X-Credits-Remaining reports the balance after each call.

For a deeper look at deduplicating retries, read Retry Web API Calls Without Paying Twice. Caching results you've already paid for also removes many repeat calls: see Cache API Responses in Python.

What does a failed call cost?

Nothing. A successful call costs the credits shown on its endpoint, which is 1 credit for every Google Search endpoint. A failed call returns X-Credits-Used: 0, so a 422 typo or a 503 costs 0 credits however many times it happens. The only way to pay for a failure is the timeout case above, where the call succeeded but you never saw it.

The arithmetic for a nightly job of 10,000 searches is simple: 10,000 calls × 1 credit = 10,000 credits, if every call succeeds. Retries that follow a 429 or 503 add nothing, because those attempts were never billed. Only a retry after a billed read timeout adds a credit. The docs cover Credits in full, and pricing shows what credits cost.

The free tier gives you 1,000 credits a month after you verify an email address, with no credit card. That's enough to run the failure paths on purpose and watch the headers.

How do you stay under the rate limit in the first place?

Limits are per account, counted per minute and shared by all your keys. Free accounts get 1 request a second (60 a minute). Paid packs run at the rate of the credits in use, and at least 5 a second (300 a minute). Every response carries RateLimit-Remaining and RateLimit-Reset, so you can slow down before a 429 rather than after.

The loop above does this in two lines: when RateLimit-Remaining reaches 0, it sleeps for RateLimit-Reset seconds. With several workers sharing one account, each worker only pauses after it sees the zero in its own response. A shared token bucket or a semaphore is the cleaner fix at higher volumes.

Retry-After is the standard HTTP header for this, defined in RFC 9110. The docs have a page on rate limits too. If you're looping over result pages, Paginate a Web Data API in Python shows how to stop safely.

Next step: get a free API key, put it in API_KEY, and run the caller above with a bad key, a bad hl value and a good query. You'll see all three paths in under a minute.

pythonerror handlingretriesrequestsrate limits