Blog / Data engineering

Retry Web API Calls Without Paying Twice

31 Jul 2026Perceptlayer10 min read

Retry only the statuses that can succeed on a second try: 429, 502 and 503. Wait a random time below a doubling cap, give up after a handful of attempts, and fail fast on everything else. On Perceptlayer a failed response costs nothing and returns X-Credits-Used: 0, so a retry after an error you received adds nothing to your bill. A timeout is different: you never saw the response, and Perceptlayer requests have no idempotency key, so a retry after a timeout can be a second paid call.

Which errors are worth retrying?

Retry an error only if the next attempt could plausibly return a different answer. A rate limit is about request pace, and the 502 and 503 codes signal trouble on the service side, so waiting and trying again is a reasonable bet. That is our recommendation, not a guarantee, which is why the loop caps attempts. A malformed parameter or a revoked key will fail identically every time, so retrying it only wastes time.

Perceptlayer errors come back as an HTTP status plus a JSON body, {"error": {"code", "message", "request_id"}}. Branch on the status or the code, and keep the request_id for any support conversation. The full list is in the errors docs.

Network errors and client timeouts belong in the retry set too, with one caveat covered below: when you never saw a response, the zero-cost rule doesn't tell you what happened.

Why does a retry not cost you twice here?

A call is debited only when it succeeds. Failed calls cost nothing and return X-Credits-Used: 0, and a successful response carries X-Credits-Used (what the call cost) and X-Credits-Remaining (the balance after it). So when you receive an error response and retry, the cost of the request is the cost of its one success: 1 credit for nearly every endpoint, however many error responses came first. The rule covers responses you actually received. It says nothing about an attempt that timed out.

That changes how you design the loop. On an API that bills per attempt, you'd weigh each retry against its price and keep the retry count low. Here the price of an extra attempt that returns an error is zero credits, so the limits you set are about time, load and your own deadline, not money. The credits docs describe the rule in full.

The arithmetic

Say a job fetches 10,000 items at 1 credit each.

• Credits spent: 10,000 calls × 1 credit = 10,000 credits, the size of one Starter pack. The pricing page shows what credits cost.
• If 800 of those calls hit a 503 first and succeed on the second attempt, you made 10,800 requests and still spent 10,000 credits.

What retries do cost is time and traffic, and time is the budget you tune. With the loop below, the worst case for one call is five attempts. Without a Retry-After header that is at most 7.5 seconds of sleeping; when the server sends one, the loop honours it up to the 30-second cap per wait, so four sleeps can reach 120 seconds. Add up to 30 seconds per attempt if each one runs into the client timeout. In the example above, the 800 retried calls add 800 round trips and their waits, not a single credit. Credits stay flat while time grows with the retry count, so you set max_attempts and cap against your deadline, not your balance.

How do you retry with exponential backoff and jitter?

Wait longer after each failure and randomise the wait. The ceiling doubles per attempt (0.5 s, 1 s, 2 s, 4 s) and the actual sleep is a random value below it, so clients rejected together don't return together. The pattern is described in AWS's Exponential Backoff and Jitter (2015). The loop below implements it; if you run several workers against one key it matters, and a single sequential script benefits less.

Pacing beats retrying. Request limits are per account, shared by all its keys, and counted per minute: free accounts get 1 request a second (60 a minute). Every response carries RateLimit-Remaining (requests left this minute) and RateLimit-Reset (seconds until the minute resets), so a worker can slow down or sleep until the reset before it draws a 429. See rate limits for paid rates.

The request

This article doesn't pick an endpoint for you, because the loop works the same on any of them. Open an API from the API list, choose an endpoint in its playground, fill in its parameters and run a real call with your key in the playground, then copy the endpoint URL for ENDPOINT_URL. Send your key in the X-API-Key header (see authentication):

curl -i "$ENDPOINT_URL" -H "X-API-Key: $API_KEY"

The -i flag prints the response headers, so you can see X-Credits-Used and X-Credits-Remaining. On a failed call you'll see X-Credits-Used: 0.

If a one-off shell script is all you need, curl can retry for you:

curl --retry 5 --retry-max-time 60 -i "$ENDPOINT_URL" -H "X-API-Key: $API_KEY"

The curl manual documents --retry and --retry-max-time; check it for which failures curl retries. It has no notion of your credit balance, so it won't stop a whole job on a 402. For anything that runs unattended, write the loop yourself.

The loop

This Python function retries only the retryable statuses, applies full jitter, and raises immediately on everything else. It also waits for the Retry-After header, which Perceptlayer sends with 429 and 503 responses (see the errors docs). It needs pip install requests.

import os
import random
import time

import requests

RETRYABLE = {429, 502, 503}


class ApiError(Exception):
    def __init__(self, status, code, request_id):
        super().__init__(f"{status} {code} (request_id={request_id})")
        self.status = status
        self.code = code
        self.request_id = request_id


def call_with_retry(url, params=None, method="GET",
                    max_attempts=5, base=0.5, cap=30.0, timeout=30):
    headers = {"X-API-Key": os.environ["API_KEY"]}
    last = None

    for attempt in range(max_attempts):
        resp = None
        try:
            resp = requests.request(method, url, params=params,
                                    headers=headers, timeout=timeout)
        except (requests.ConnectionError, requests.Timeout):
            last = ApiError(0, "network_error", None)

        if resp is not None:
            if resp.ok:
                return resp
            try:
                err = resp.json().get("error", {})
            except ValueError:
                err = {}
            last = ApiError(resp.status_code, err.get("code"), err.get("request_id"))
            if resp.status_code not in RETRYABLE:
                raise last  # 401, 402, 403, 404, 422: retrying cannot help

        if attempt == max_attempts - 1:
            break

        wait = random.uniform(0, min(cap, base * 2 ** attempt))
        retry_after = resp.headers.get("Retry-After") if resp is not None else None
        if retry_after and retry_after.isdigit():
            wait = max(wait, min(cap, int(retry_after)))
        time.sleep(wait)

    raise last

Three details are worth a second look. The loop raises on a non-retryable status before it sleeps, so a bad key fails in one round trip. The Retry-After branch uses the wait Perceptlayer sends with a 429 or 503, and falls back to jittered backoff when a response has none. This code reads only a plain number of seconds and ignores any other form of the header. And the final raise last surfaces the last real error, with its request_id, rather than a generic "retries exhausted".

The parameters

With these defaults the delay ceilings are 0.5, 1, 2 and 4 seconds, so a call that never recovers and gets no Retry-After header gives up after at most 7.5 seconds of waiting plus the time of the five attempts themselves. If the server sends Retry-After, each wait can rise to the 30-second cap, which makes the worst case 120 seconds of sleeping.

What should you do when a call times out?

A timeout means you never saw the response, so you don't know how that attempt ended. The zero-cost rule applies to failed responses you receive. If the request succeeded on the server and the reply was lost, it may have been debited, and Perceptlayer requests have no idempotency key that would stop the retry from being debited again. Retry anyway, with the same cap on attempts, but check the balance rather than assume.

The check uses X-Credits-Remaining. Between two consecutive successes from a single sequential worker, the balance should drop by exactly the second call's X-Credits-Used. A bigger drop means a debit you didn't see, most likely a timed-out attempt that went through. To cover the first call too, read a starting balance before the loop: error responses carry X-Credits-Remaining as well, so a request with no parameters, which the endpoint should reject, gives you the baseline for free.

import os

url = os.environ["ENDPOINT_URL"]
baseline = requests.get(url, headers={"X-API-Key": os.environ["API_KEY"]}, timeout=30)
previous = int(baseline.headers["X-Credits-Remaining"])

spent = 0
for item in items:
    resp = call_with_retry(url, params={"q": item})
    used = int(resp.headers["X-Credits-Used"])
    remaining = int(resp.headers["X-Credits-Remaining"])
    spent += used
    if previous - remaining != used:
        print(f"unexpected debit: balance fell {previous - remaining}, call cost {used}")
    previous = remaining

print(f"spent {spent} credits on confirmed calls, {previous} remaining")

This snippet continues from the loop above and reuses its requests import. If the endpoint accepts an empty request, the baseline call succeeds and is billed like any other, so pick an endpoint with a required parameter.

Two limits on this check. It only works when nothing else spends from the same key at the same time, so don't run it alongside a second worker or the playground. And it detects extra debits after the fact; it doesn't prevent them. The params here are placeholders: use the parameter names the playground shows for your endpoint. Read X-Credits-Used from the response, not from a constant, so the check stays right on an endpoint that costs more than 1 credit.

How do you stop a retry loop from running away?

Bound it three ways: attempts per call, wait per attempt, and failures across the whole job. The first two are the max_attempts and cap parameters. The third is a circuit breaker. If every call in a window fails with 503, more retries won't help, and stopping early and resuming later is cheaper than hammering a service that is already down.

A simple version counts consecutive exhausted calls. After a few in a row, pause the job for a minute or two, or exit and let your scheduler start it again. For 402, don't pause at all: the balance is empty, there is no automatic top-up, and every further call will fail until you add credits. Raise an alert and stop. If the same calls repeat across a job, caching responses removes some of them before they can fail, and safe page loops cover the other common long-running pattern. More like this is in the data engineering category.

Can you retry through an AI agent or MCP tool?

Yes, and the cost rule is the same. The call_endpoint tool on the MCP server spends the endpoint's credits on success only, and its result reports the credits used and remaining. search_endpoints and get_endpoint look up the catalogue and cost nothing. An agent that retries a failed call_endpoint therefore pays once, on the attempt that works.

Put the retry policy in the tool wrapper or the agent's instructions, not in the model's judgment. Tell it which errors to retry, how many times, and to stop on insufficient_credits. A model left to improvise will sometimes retry a 422 with the same parameters. The MCP docs show how to connect the server.

Next step

Get a free API key: 1,000 free credits every month after you verify an email address, no credit card. Pick an endpoint in the playground on any API page, paste its URL into the loop above and run it. Check the headers on the first response, then send a request with a deliberately bad parameter and confirm it fails once with X-Credits-Used: 0 and no retries. What credits cost is on the pricing page.

retriesbackoffrate limitscreditserror handling