Blog / Data engineering

Paginate a Web Data API in Python: Safe Page Loops

6 Aug 2026Perceptlayer8 min read

To paginate a web data API in Python, request page=1, read the has_more flag in the response, and keep going until it turns false. Add three guards: a page cap, a deduplication set and a credit ceiling. This guide does it against the search endpoint of the eBay API, where every page is one call and one credit, so a full pull has a cost you can calculate before you run it.

How does pagination work on the eBay search endpoint?

The search endpoint is page-based. You send q with page and per_page, and the response tells you where you are (page, per_page), whether another page exists (has_more) and how many results match overall (total_results). The endpoint's parameter list has no cursor and the recorded response returns none, so there's nothing to carry between calls. You increment page yourself and let has_more say when to stop.

Here's the first page as a cURL call:

curl "https://api.perceptlayer.com/v1/ebay/search?q=coffee+maker&page=1&per_page=60" \
  -H "X-API-Key: $API_KEY"

Page-number paging is simple, but it leaves the safety to you. Nothing in the protocol stops a loop that runs until your balance is gone, which is why the rest of this article is about stop conditions. The full parameter list for the endpoint is on the eBay Search API page, and the key setup is in the authentication docs.

Which parameters control paging?

Two parameters control paging: page selects which page you get and per_page sets how many results each page holds, from three allowed values. Everything else on the endpoint (filters and sorting) narrows or reorders the result set, and you should keep those identical across every page of one pull. Changing a filter mid-pull gives you pages from two different result sets.

The docs don't state a default page size or a maximum page depth, so don't assume either. Pass per_page explicitly and let the stop conditions find the end. The eBay search results guide covers the filters in more detail.

What does a result page look like?

A page is a JSON object with success, and a data object holding the paging fields next to a results array. Each result carries a product_id, a title, a link, a price object and a few optional fields. The recorded response for q=coffee maker, page=1, per_page=60 looks like this, trimmed:

{
  "success": true,
  "data": {
    "query": "coffee maker",
    "page": 1,
    "per_page": 60,
    "has_more": true,
    "results": [
      {
        "product_id": "366702966104",
        "title": "Mr. Coffee 4-in-1 Single-Serve Latte, Iced, and Hot Coffee",
        "link": "https://www.ebay.com/itm/366702966104",
        "price": {
          "raw": "$60.00",
          "extracted": 60,
          "currency": "USD"
        },
        "sponsored": true,
        "location": "United States"
      },
      …
    ],
    "total_results": 110000,
    "related_searches": […],
    "categories": […]
  }
}

Three fields matter for pagination. has_more is your primary stop signal. total_results (110000 for this query) lets you estimate the number of pages before you start. product_id is used here as the dedupe key. Fields such as subtitle, thumbnail, sponsored and location are optional extras, so read them with .get() rather than indexing.

How do you fetch every page safely?

Write the loop around its exit conditions rather than around the happy path. A safe pagination loop stops on the first of five conditions: has_more is false, a page returns no results, a page contributes no new items, the page cap is reached, or the credits spent reach a ceiling. It also stops if X-Credits-Remaining shows the balance can't pay for another page, and it returns the reason it stopped, so a partial pull is never mistaken for a complete one.

Free accounts get 1 request a second, so a tight loop with no pause can hit a 429. The code below sleeps briefly between pages and honours Retry-After when a 429 or 503 carries it. Retries and billing are covered in retrying without paying twice.

import os
import time

import requests

BASE = "https://api.perceptlayer.com"
HEADERS = {"X-API-Key": os.environ["API_KEY"]}
RETRYABLE = {429, 502, 503}


def get_page(params, max_attempts=4):
    """GET one page, or None if the network kept failing.

    A 429, 502 or 503 is a failed call and isn't billed, so retrying it is free.
    A timeout is different: there's no idempotency key, so the request may have
    succeeded and been billed even though no reply arrived. Retrying one can
    cost a second credit.
    """
    r = None
    for attempt in range(max_attempts):
        try:
            r = requests.get(
                f"{BASE}/v1/ebay/search", params=params, headers=HEADERS, timeout=30
            )
        except requests.RequestException:
            r = None  # timeout or connection error: retry after a pause
        if (r is None or r.status_code in RETRYABLE) and attempt < max_attempts - 1:
            delay = 2**attempt
            if r is not None:
                try:  # 429 and 503 carry Retry-After in seconds
                    delay = max(delay, float(r.headers["Retry-After"]))
                except (KeyError, ValueError):
                    pass
            time.sleep(delay)
            continue
        return r
    return r


def collect(q, per_page=240, max_pages=50, credit_ceiling=25):
    seen = {}  # product_id -> result
    spent = 0
    reason = "page cap reached"

    for page in range(1, max_pages + 1):
        if spent >= credit_ceiling:
            reason = "credit ceiling reached"
            break

        r = get_page({"q": q, "page": page, "per_page": per_page})
        if r is None:
            reason = f"network error on page {page}"
            break
        if r.status_code != 200:
            try:
                code = r.json().get("error", {}).get("code", "unknown")
            except ValueError:  # non-JSON body, e.g. from a 502 or 503
                code = "unknown"
            reason = f"error {r.status_code} {code} on page {page}"
            break

        cost = int(r.headers.get("X-Credits-Used", 0))
        spent += cost
        remaining = int(r.headers.get("X-Credits-Remaining", -1))
        data = r.json()["data"]
        results = data["results"]

        new = 0
        for item in results:
            if item["product_id"] not in seen:
                seen[item["product_id"]] = item
                new += 1

        if not data["has_more"]:
            reason = "has_more is false"
            break
        if not results:
            reason = "empty page"
            break
        if new == 0:
            reason = "page added no new items"
            break
        if 0 <= remaining < cost:
            reason = "balance below the next page's cost"
            break
        time.sleep(1)  # free accounts get 1 request a second

    return list(seen.values()), spent, reason


items, spent, reason = collect("coffee maker")
print(f"{len(items)} unique items, {spent} credits, stopped: {reason}")

The loop checks credit_ceiling before each call, so it never starts a page it has already decided not to pay for. It reads X-Credits-Used from the response instead of assuming 1 credit, so if an endpoint's price ever changes, the ceiling still holds. It also reads X-Credits-Remaining and stops when the balance is below what the last page cost, assuming the next page costs the same. For more on credits, see the credits docs.

How do you deduplicate results across pages?

Key every result on product_id and keep the first copy you see. The docs don't say whether the result set stays fixed while you page through it, so don't assume it does. If it shifts between your calls, an item near a page boundary could show up twice. A dictionary keyed on product_id makes that harmless, and it gives you the "page added no new items" stop signal for free.

new = 0
for item in results:
    if item["product_id"] not in seen:
        seen[item["product_id"]] = item
        new += 1

Two details are worth keeping. First, keep the first copy, not the last, so a result's position reflects the order you first received it. Second, treat the new == 0 exit as a safety net. It catches a loop that keeps getting the same page, for example because a parameter was dropped from the request, so the loop ends instead of running to its page cap.

Sponsored results carry "sponsored": true. If you only want organic listings, filter on that field after deduplication, not before, so your stop logic still sees every item.

What does a full pull cost?

Every successful page costs 1 credit, so a pull of N pages costs N credits. Failed calls cost nothing and return X-Credits-Used: 0. Larger pages mean fewer calls for the same number of results, so per_page=240 is the cheapest setting per result. The table states each pull in credits; see pricing for what credits cost.

The last two rows are a calculation on the total_results value in the recorded response, not a recorded pull. They also assume every page is full, at 240 and 60 results; only a 60-result page is recorded, so the real page count may differ. The arithmetic: 110,000 ÷ 240 = 458.3, so 459 pages × 1 credit = 459 credits. At 60 per page, 110,000 ÷ 60 = 1,833.3, so 1,834 pages × 1 credit = 1,834 credits. Either pull fits within the smallest pack's 10,000 credits.

The default collect("coffee maker") call stops at its 50-page cap, so to attempt the 459-page calculation, raise both guards just above the figure:

items, spent, reason = collect(
    "coffee maker", per_page=240, max_pages=500, credit_ceiling=500
)

Treat the 110,000 rows as a ceiling, not a promise: a full 459-page pull may not be possible. total_results is the figure from the recorded coffee maker response, and the docs don't say how deep you can page. Your has_more check finds the real end. The free tier's 1,000 credits a month cover 459 pages more than twice over. For a wider view of budgeting, see capping web data spend. More pipeline guides are in the data engineering category.

How do you handle errors and retries mid-pull?

Retry the transient errors, stop on the permanent ones, and keep what you've already collected. Errors come back as an HTTP status and a JSON body with error.code, error.message and error.request_id. A 429, 502 or 503 is a failed call and isn't billed, so retrying it costs nothing extra beyond the wait. A client timeout is the exception: there's no idempotency key, so a request whose reply was lost may have succeeded and been billed, and retrying it can cost a second credit. The retry guide goes deeper.

The get_page function above retries the first three, and on network errors too. It backs off 1, 2, then 4 seconds, or waits longer when a Retry-After header asks for it, the standard response to 429 Too Many Requests. Everything else falls through to collect, which records the reason and returns the items it already has.

For long pulls, save seen to disk every few pages, and cache pages you've already fetched so a rerun doesn't pay for them again (see caching API responses). Then a 402 or a crashed process costs you the remaining pages, not the whole run. Resume by starting from the last saved page number and loading the saved product_id values back into seen. For more, see the errors docs and the rate limit docs.

If you're calling the API through the requests library as above, the Requests documentation covers sessions, which reuse connections across a pull of hundreds of pages.

Get a free API key at /signup, set API_KEY, and run collect with a credit_ceiling of 20 to see the whole loop end to end for 20 credits or fewer.

paginationpythonebaycreditsdata engineering