--- title: "Handle Web Data API Errors in Python: 4xx, 5xx and Timeouts" description: "Map the API's error codes to retry, fix or stop in Python: 429 and 503 honour Retry-After, 4xx stops, and failed calls cost 0 credits." url: https://perceptlayer.com/blog/handle-web-data-api-errors-python --- [Blog](https://perceptlayer.com/blog) / [Data engineering](https://perceptlayer.com/blog/category/data-engineering) # Handle Web Data API Errors in Python: 4xx, 5xx and Timeouts 22 Sept 2026 Perceptlayer 7 min read Sort every failed response into one of three actions. Retry `429`, `503` and `502` with a wait. Fix the request on `401`, `403`, `404` and `422`. Stop the job on `402`. Timeouts are the odd case: they carry no status code, and the call may have succeeded. This guide gives you the mapping and a Python function with the `requests` library that applies it. ## Key takeaways - Every error returns an HTTP status and a JSON body with `error.code`, `error.message` and `error.request_id`. Branch on the code, not the message. - `429` and `503` carry a `Retry-After` header in seconds. Wait that long, then retry. - Failed calls cost nothing and return `X-Credits-Used: 0`. A client-side timeout is the exception to plan for, because the call may have been billed. - `invalid_parameters` (422), `invalid_api_key` (401) and `not_found` (404) never fix themselves. Retrying them wastes time and rate limit. - `insufficient_credits` (402) means the prepaid balance is empty. Nothing tops it up automatically, so stop the job. ## Which API errors should you retry, fix or stop on? Going by the code names, retry the ones about the moment ( `rate_limited`, `temporarily_unavailable`, `source_error`). Fix the ones about your request ( `invalid_api_key`, `restricted_endpoint`, `not_found`, `invalid_parameters`). Stop on the one about your account ( `insufficient_credits`). The table adds the action we recommend to each code, and every row costs 0 credits. For what each code covers, see [Errors](https://docs.perceptlayer.com/errors) in the docs. | Status | `error.code` | Action | | --- | --- | --- | | 401 | `invalid_api_key` | Fix: check `X-API-Key` | | 402 | `insufficient_credits` | Stop: the prepaid balance is empty, so buy credits, then rerun | | 403 | `restricted_endpoint` | Fix: accept the restricted-endpoint terms in the dashboard | | 404 | `not_found` | Fix: check the path | | 422 | `invalid_parameters` | Fix: check the parameters | | 429 | `rate_limited` | Retry after `Retry-After` seconds (limits are counted per minute) | | 502 | `source_error` | Retry with backoff, capped | | 503 | `temporarily_unavailable` | Retry after `Retry-After` seconds | The 502 row has no documented `Retry-After`, so the code below falls back to exponential backoff with jitter. Any other 5xx isn't on the documented list. Treating it as retryable with a hard cap is our policy, not a documented behaviour, and the cap is what keeps that safe. ## How do you call the endpoint and read the response? Send the key in the `X-API-Key` header and the parameters in the query string. The examples use the [Google Search API](https://perceptlayer.com/apis/google-search) `search` endpoint, which costs 1 credit per successful call. The same pattern works for any of the 1,659 endpoints across 248 websites, because every error uses the same body shape. ```bash curl "https://api.perceptlayer.com/v1/google-search/search?query=best+restaurants+in+Berlin&hl=en&safe=off" \ -H "X-API-Key: $API_KEY" ``` These are the parameters that matter for error handling. `query` is the only required one. Check your values against the documented limits before you send them. The full list is on the [Search endpoint page](https://perceptlayer.com/apis/google-search/search). | Parameter | Type | Required | Documented limit | | --- | --- | --- | --- | | `query` | string | yes | none stated | | `hl` | string | no | up to 2 characters | | `gl` | string | no | up to 2 characters | | `safe` | string | no | `active` or `off` | | `device` | string | no | `desktop`, `tablet` or `mobile` | | `page` | integer | no | none stated | A successful call returns JSON with named fields. This is an excerpt of a recorded response for the request above: ```json { "search_information": { "query_displayed": "best restaurants in Berlin", "organic_results_count": 6 }, "organic_results": [ { "position": 1, "title": "The Best Restaurants in Berlin - Food Guide & Map", … } ], … } ``` To see an error on purpose, send a wrong key with `curl -i`. Read the status and the `error.code`, and check the headers: errors carry `X-Credits-Remaining`, and failed calls return `X-Credits-Used: 0`. The `-i` flag prints the headers, which is where `Retry-After` and the `RateLimit-*` values also show up. ## How do you write the retry loop in Python? Write one function that every call goes through. It raises a custom exception for errors you must fix or stop on, retries the rest a fixed number of times, and sleeps for `Retry-After` when the header is there. It sets a timeout on every call, because `requests` waits forever without one ( [Requests documentation](https://requests.readthedocs.io/en/latest/user/advanced/#timeouts)). ```python import logging import os import random import time import requests BASE_URL = "https://api.perceptlayer.com" RETRY_CODES = {"rate_limited", "temporarily_unavailable", "source_error"} log = logging.getLogger("webdata") class ApiError(Exception): """A response you must fix or stop on. Retrying won't help.""" def __init__(self, status, code, message, request_id): super().__init__(f"{status} {code}: {message} (request_id={request_id})") self.status = status self.code = code self.request_id = request_id class RetriesExhausted(Exception): pass def parse_error(resp): try: err = resp.json()["error"] return err["code"], err["message"], err["request_id"] except (ValueError, KeyError, TypeError): return "unknown", "no error body", None def wait_seconds(retry_after, attempt): try: return min(float(retry_after), 60) except (TypeError, ValueError): return min(2 ** attempt, 30) + random.random() def call(path, params, attempts=4, timeout=(5, 30)): headers = {"X-API-Key": os.environ["API_KEY"]} for attempt in range(1, attempts + 1): retry_after = None try: resp = requests.get( BASE_URL + path, params=params, headers=headers, timeout=timeout ) except requests.exceptions.ReadTimeout: # The request was sent. It may have succeeded and been billed. reason = "read timeout" except requests.exceptions.ConnectionError: # Includes connect timeouts: the request never reached the API. reason = "connection failed" else: if resp.ok: if resp.headers.get("RateLimit-Remaining") == "0": time.sleep(float(resp.headers.get("RateLimit-Reset", 1))) return resp.json() code, message, request_id = parse_error(resp) if code not in RETRY_CODES and resp.status_code < 500: raise ApiError(resp.status_code, code, message, request_id) reason = f"{resp.status_code} {code} (request_id={request_id})" retry_after = resp.headers.get("Retry-After") log.warning("attempt %d of %d failed: %s", attempt, attempts, reason) if attempt < attempts: time.sleep(wait_seconds(retry_after, attempt)) raise RetriesExhausted(f"gave up after {attempts} attempts: {reason}") ``` The function never logs a response body, only the status, code and `request_id`. Keep the `request_id`: it's what you quote if you ask for help with a specific failure. Here's how a caller uses it: ```python try: data = call( "/v1/google-search/search", {"query": "best restaurants in Berlin", "hl": "en", "safe": "off"}, ) print(data["search_information"]["organic_results_count"]) except ApiError as err: if err.code == "insufficient_credits": raise SystemExit("Out of credits. Buy a pack, then rerun.") print(f"Fix the request: {err}") except RetriesExhausted as err: print(f"Still failing, try again later: {err}") ``` ### Why not just use urllib3's Retry? You can. `requests` sits on urllib3, and its [`Retry` class](https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Retry) mounts on a session with `status_forcelist` and `respect_retry_after_header`. It can't read `error.code` from the body or log the `request_id`. If you only need retries, use `Retry`. For the mapping above, the explicit loop is a few dozen lines. ## How do you handle timeouts without paying twice? A connect timeout is safe to retry because the request never reached the API. A read timeout isn't: the API may have finished the call and billed it, and your client just never saw the reply. Requests have no idempotency key, so the API can't tell a retry from a new call. Set both timeouts, retry with a cap, and accept that a retry after a read timeout can cost one extra credit. The tuple `timeout=(5, 30)` in the code above means 5 seconds to connect and 30 to read. Pick the read value from what you see in your own traffic, then add headroom. Too tight and you turn slow successes into duplicate charges. Too loose and a stuck call holds a worker. Two habits keep the extra credits small. Cap attempts: four attempts means at most three retries. And log read timeouts separately, because a rising count tells you the timeout is too tight. If you need to reconcile, `X-Credits-Remaining` reports the balance after each call. For a deeper look at deduplicating retries, read [Retry Web API Calls Without Paying Twice](https://perceptlayer.com/blog/retry-web-api-calls-without-paying-twice). Caching results you've already paid for also removes many repeat calls: see [Cache API Responses in Python](https://perceptlayer.com/blog/cache-web-data-api-responses). ## What does a failed call cost? Nothing. A successful call costs the credits shown on its endpoint, which is 1 credit for every Google Search endpoint. A failed call returns `X-Credits-Used: 0`, so a `422` typo or a `503` costs 0 credits however many times it happens. The only way to pay for a failure is the timeout case above, where the call succeeded but you never saw it. The arithmetic for a nightly job of 10,000 searches is simple: 10,000 calls × 1 credit = 10,000 credits, if every call succeeds. Retries that follow a `429` or `503` add nothing, because those attempts were never billed. Only a retry after a billed read timeout adds a credit. The docs cover [Credits](https://docs.perceptlayer.com/credits) in full, and [pricing](https://perceptlayer.com/pricing) shows what credits cost. The free tier gives you 1,000 credits a month after you verify an email address, with no credit card. That's enough to run the failure paths on purpose and watch the headers. ## How do you stay under the rate limit in the first place? Limits are per account, counted per minute and shared by all your keys. Free accounts get 1 request a second (60 a minute). Paid packs run at the rate of the credits in use, and at least 5 a second (300 a minute). Every response carries `RateLimit-Remaining` and `RateLimit-Reset`, so you can slow down before a `429` rather than after. The loop above does this in two lines: when `RateLimit-Remaining` reaches `0`, it sleeps for `RateLimit-Reset` seconds. With several workers sharing one account, each worker only pauses after it sees the zero in its own response. A shared token bucket or a semaphore is the cleaner fix at higher volumes. `Retry-After` is the standard HTTP header for this, defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-retry-after). The docs have a page on [rate limits](https://docs.perceptlayer.com/rate-limits) too. If you're looping over result pages, [Paginate a Web Data API in Python](https://perceptlayer.com/blog/paginate-web-data-api-python) shows how to stop safely. Next step: [get a free API key](https://perceptlayer.com/signup), put it in `API_KEY`, and run the caller above with a bad key, a bad `hl` value and a good query. You'll see all three paths in under a minute. ## APIs in this post - [Google Search **1 credit per call** · 7 endpoints](https://perceptlayer.com/apis/google-search) ## Frequently asked questions ### Do failed API calls cost credits? No. Failed calls cost nothing and return `X-Credits-Used: 0`, along with `X-Credits-Remaining` so you can see the balance. The one case that can bill is a client timeout: if the reply was lost, the call may have succeeded and been billed, and requests have no idempotency key to prevent a duplicate. ### How long should I wait after a 429 or 503? Wait the number of seconds in the `Retry-After` header. Both `429` ( `rate_limited`) and `503` ( `temporarily_unavailable`) carry it. If the header is missing, fall back to exponential backoff with jitter and a cap, as the `wait_seconds` function does. A `502` has no documented `Retry-After`, so it uses the backoff. ### Should I retry a 422 invalid\_parameters error? No. `invalid_parameters` is about the parameters you sent, so the same request fails again. Compare them with the documented limits, such as `hl` at up to 2 characters or `safe` as `active` or `off`, and fix the value. The endpoint page lists each parameter's type and allowed values. ### What happens when insufficient\_credits (402) appears? The prepaid balance is empty. Calls return `402` and nothing more is debited. There's no automatic top-up, so the job should stop and tell a person. Purchased credits last 12 months, and the free tier adds 1,000 credits each month. See [pricing](https://perceptlayer.com/pricing) for what a pack costs. ### Does the same cost rule apply through MCP? Yes. The MCP server at `https://api.perceptlayer.com/mcp` exposes `call_endpoint`, which spends its credits on success only and reports the credits used and remaining. A failed tool call costs nothing, as over HTTP. The [AI agents docs](https://docs.perceptlayer.com/ai-agents) cover the setup. python error handling retries requests rate limits ## More from the blog 1. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 31 Jul 2026 Perceptlayer 10 min read ## [Retry Web API Calls Without Paying Twice](https://perceptlayer.com/blog/retry-web-api-calls-without-paying-twice) Retry only 429, 502 and 503. Failed responses return X-Credits-Used: 0, but a timeout is the gap. The credit arithmetic, a Python loop and a balance check. 2. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 19 Oct 2026 Perceptlayer 7 min read ## [Concurrent API Requests in Python: Asyncio Under Rate Limits](https://perceptlayer.com/blog/concurrent-api-requests-python-rate-limit) A Python asyncio fetcher that paces calls to your per-second limit, honours Retry-After and prices a 20,000-call batch at 20,000 credits. 3. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 20 Oct 2026 Perceptlayer 6 min read ## [Detect Schema Changes in API JSON Responses](https://perceptlayer.com/blog/web-data-api-json-schema-changes) Infer a path-and-type schema from a recorded Google Search response (71 paths), diff each new response against it and alert on removed or retyped fields.