--- title: "Paginate a Web Data API in Python: Safe Page Loops" description: "Page through the eBay search endpoint in Python with has_more, dedupe by product_id and a credit ceiling. Each page costs 1 credit." url: https://perceptlayer.com/blog/paginate-web-data-api-python --- [Blog](https://perceptlayer.com/blog) / [Data engineering](https://perceptlayer.com/blog/category/data-engineering) # Paginate a Web Data API in Python: Safe Page Loops 6 Aug 2026 Perceptlayer 8 min read To paginate a web data API in Python, request `page=1`, read the `has_more` flag in the response, and keep going until it turns false. Add three guards: a page cap, a deduplication set and a credit ceiling. This guide does it against the search endpoint of the [eBay API](https://perceptlayer.com/apis/ebay), where every page is one call and one credit, so a full pull has a cost you can calculate before you run it. ## Key takeaways - `GET /v1/ebay/search` pages with `page` and `per_page` (60, 120 or 240), and each response reports `has_more` and `total_results`. - Stop on the first of five conditions: `has_more` is false, a page comes back empty, a page adds no new items, your page cap is reached, or your credit ceiling is reached. - Deduplicate on `product_id`. The docs don't promise a stable result set between calls, so guard against repeats instead of assuming none. - Each successful page costs 1 credit and failed calls cost 0, so retrying a 429, 502 or 503 costs nothing extra. A request that timed out client-side may still have been billed, because there's no idempotency key. - Choosing `per_page=240` over 60 cuts the number of calls, and therefore the credits, for the same result set. ## How does pagination work on the eBay search endpoint? The search endpoint is page-based. You send `q` with `page` and `per_page`, and the response tells you where you are ( `page`, `per_page`), whether another page exists ( `has_more`) and how many results match overall ( `total_results`). The endpoint's parameter list has no cursor and the recorded response returns none, so there's nothing to carry between calls. You increment `page` yourself and let `has_more` say when to stop. Here's the first page as a cURL call: ```bash curl "https://api.perceptlayer.com/v1/ebay/search?q=coffee+maker&page=1&per_page=60" \ -H "X-API-Key: $API_KEY" ``` Page-number paging is simple, but it leaves the safety to you. Nothing in the protocol stops a loop that runs until your balance is gone, which is why the rest of this article is about stop conditions. The full parameter list for the endpoint is on the [eBay Search API page](https://perceptlayer.com/apis/ebay/search), and the key setup is in the [authentication docs](https://docs.perceptlayer.com/authentication). ## Which parameters control paging? Two parameters control paging: `page` selects which page you get and `per_page` sets how many results each page holds, from three allowed values. Everything else on the endpoint (filters and sorting) narrows or reorders the result set, and you should keep those identical across every page of one pull. Changing a filter mid-pull gives you pages from two different result sets. | Parameter | Type | Required | Notes | | --- | --- | --- | --- | | `q` | string | yes | Up to 255 characters | | `page` | integer | no | The page to return | | `per_page` | integer | no | One of `60`, `120`, `240` | | `sort` | integer | no | One of `1`, `7`, `10`, `12`, `15`, `16` | | `category` | integer | no | Narrows the result set | | `price_min`, `price_max` | number | no | Price filters | | `buy_it_now`, `auction`, `free_shipping` | boolean | no | Listing filters | | `condition[]` | list of integer | no | Condition filter | | `preferred_location`, `location_postal` | integer, string | no | Location filters; `location_postal` up to 20 characters | The docs don't state a default page size or a maximum page depth, so don't assume either. Pass `per_page` explicitly and let the stop conditions find the end. The [eBay search results guide](https://perceptlayer.com/blog/ebay-item-search-data-json) covers the filters in more detail. ## What does a result page look like? A page is a JSON object with `success`, and a `data` object holding the paging fields next to a `results` array. Each result carries a `product_id`, a `title`, a `link`, a `price` object and a few optional fields. The recorded response for `q=coffee maker`, `page=1`, `per_page=60` looks like this, trimmed: ```json { "success": true, "data": { "query": "coffee maker", "page": 1, "per_page": 60, "has_more": true, "results": [ { "product_id": "366702966104", "title": "Mr. Coffee 4-in-1 Single-Serve Latte, Iced, and Hot Coffee", "link": "https://www.ebay.com/itm/366702966104", "price": { "raw": "$60.00", "extracted": 60, "currency": "USD" }, "sponsored": true, "location": "United States" }, … ], "total_results": 110000, "related_searches": […], "categories": […] } } ``` Three fields matter for pagination. `has_more` is your primary stop signal. `total_results` (110000 for this query) lets you estimate the number of pages before you start. `product_id` is used here as the dedupe key. Fields such as `subtitle`, `thumbnail`, `sponsored` and `location` are optional extras, so read them with `.get()` rather than indexing. ## How do you fetch every page safely? Write the loop around its exit conditions rather than around the happy path. A safe pagination loop stops on the first of five conditions: `has_more` is false, a page returns no results, a page contributes no new items, the page cap is reached, or the credits spent reach a ceiling. It also stops if `X-Credits-Remaining` shows the balance can't pay for another page, and it returns the reason it stopped, so a partial pull is never mistaken for a complete one. Free accounts get 1 request a second, so a tight loop with no pause can hit a 429. The code below sleeps briefly between pages and honours `Retry-After` when a 429 or 503 carries it. Retries and billing are covered in [retrying without paying twice](https://perceptlayer.com/blog/retry-web-api-calls-without-paying-twice). ```python import os import time import requests BASE = "https://api.perceptlayer.com" HEADERS = {"X-API-Key": os.environ["API_KEY"]} RETRYABLE = {429, 502, 503} def get_page(params, max_attempts=4): """GET one page, or None if the network kept failing. A 429, 502 or 503 is a failed call and isn't billed, so retrying it is free. A timeout is different: there's no idempotency key, so the request may have succeeded and been billed even though no reply arrived. Retrying one can cost a second credit. """ r = None for attempt in range(max_attempts): try: r = requests.get( f"{BASE}/v1/ebay/search", params=params, headers=HEADERS, timeout=30 ) except requests.RequestException: r = None # timeout or connection error: retry after a pause if (r is None or r.status_code in RETRYABLE) and attempt < max_attempts - 1: delay = 2**attempt if r is not None: try: # 429 and 503 carry Retry-After in seconds delay = max(delay, float(r.headers["Retry-After"])) except (KeyError, ValueError): pass time.sleep(delay) continue return r return r def collect(q, per_page=240, max_pages=50, credit_ceiling=25): seen = {} # product_id -> result spent = 0 reason = "page cap reached" for page in range(1, max_pages + 1): if spent >= credit_ceiling: reason = "credit ceiling reached" break r = get_page({"q": q, "page": page, "per_page": per_page}) if r is None: reason = f"network error on page {page}" break if r.status_code != 200: try: code = r.json().get("error", {}).get("code", "unknown") except ValueError: # non-JSON body, e.g. from a 502 or 503 code = "unknown" reason = f"error {r.status_code} {code} on page {page}" break cost = int(r.headers.get("X-Credits-Used", 0)) spent += cost remaining = int(r.headers.get("X-Credits-Remaining", -1)) data = r.json()["data"] results = data["results"] new = 0 for item in results: if item["product_id"] not in seen: seen[item["product_id"]] = item new += 1 if not data["has_more"]: reason = "has_more is false" break if not results: reason = "empty page" break if new == 0: reason = "page added no new items" break if 0 <= remaining < cost: reason = "balance below the next page's cost" break time.sleep(1) # free accounts get 1 request a second return list(seen.values()), spent, reason items, spent, reason = collect("coffee maker") print(f"{len(items)} unique items, {spent} credits, stopped: {reason}") ``` The loop checks `credit_ceiling` before each call, so it never starts a page it has already decided not to pay for. It reads `X-Credits-Used` from the response instead of assuming 1 credit, so if an endpoint's price ever changes, the ceiling still holds. It also reads `X-Credits-Remaining` and stops when the balance is below what the last page cost, assuming the next page costs the same. For more on credits, see the [credits docs](https://docs.perceptlayer.com/credits). ## How do you deduplicate results across pages? Key every result on `product_id` and keep the first copy you see. The docs don't say whether the result set stays fixed while you page through it, so don't assume it does. If it shifts between your calls, an item near a page boundary could show up twice. A dictionary keyed on `product_id` makes that harmless, and it gives you the "page added no new items" stop signal for free. ```python new = 0 for item in results: if item["product_id"] not in seen: seen[item["product_id"]] = item new += 1 ``` Two details are worth keeping. First, keep the first copy, not the last, so a result's position reflects the order you first received it. Second, treat the `new == 0` exit as a safety net. It catches a loop that keeps getting the same page, for example because a parameter was dropped from the request, so the loop ends instead of running to its page cap. Sponsored results carry `"sponsored": true`. If you only want organic listings, filter on that field after deduplication, not before, so your stop logic still sees every item. ## What does a full pull cost? Every successful page costs 1 credit, so a pull of N pages costs N credits. Failed calls cost nothing and return `X-Credits-Used: 0`. Larger pages mean fewer calls for the same number of results, so `per_page=240` is the cheapest setting per result. The table states each pull in credits; see [pricing](https://perceptlayer.com/pricing) for what credits cost. | Pull | Pages | Calculation | Credits | | --- | --- | --- | --- | | 10 pages | 10 | 10 calls × 1 credit | 10 | | 50 pages (the default cap above) | 50 | 50 calls × 1 credit | 50 | | Calculation: `total_results` of 110,000 at `per_page=240` | 459 | 459 calls × 1 credit | 459 | | Calculation: `total_results` of 110,000 at `per_page=60` | 1,834 | 1,834 calls × 1 credit | 1,834 | The last two rows are a calculation on the `total_results` value in the recorded response, not a recorded pull. They also assume every page is full, at 240 and 60 results; only a 60-result page is recorded, so the real page count may differ. The arithmetic: 110,000 ÷ 240 = 458.3, so 459 pages × 1 credit = 459 credits. At 60 per page, 110,000 ÷ 60 = 1,833.3, so 1,834 pages × 1 credit = 1,834 credits. Either pull fits within the smallest pack's 10,000 credits. The default `collect("coffee maker")` call stops at its 50-page cap, so to attempt the 459-page calculation, raise both guards just above the figure: ```python items, spent, reason = collect( "coffee maker", per_page=240, max_pages=500, credit_ceiling=500 ) ``` Treat the 110,000 rows as a ceiling, not a promise: a full 459-page pull may not be possible. `total_results` is the figure from the recorded `coffee maker` response, and the docs don't say how deep you can page. Your `has_more` check finds the real end. The free tier's 1,000 credits a month cover 459 pages more than twice over. For a wider view of budgeting, see [capping web data spend](https://perceptlayer.com/blog/ai-agent-web-data-costs). More pipeline guides are in the [data engineering category](https://perceptlayer.com/blog/category/data-engineering). ## How do you handle errors and retries mid-pull? Retry the transient errors, stop on the permanent ones, and keep what you've already collected. Errors come back as an HTTP status and a JSON body with `error.code`, `error.message` and `error.request_id`. A 429, 502 or 503 is a failed call and isn't billed, so retrying it costs nothing extra beyond the wait. A client timeout is the exception: there's no idempotency key, so a request whose reply was lost may have succeeded and been billed, and retrying it can cost a second credit. The [retry guide](https://perceptlayer.com/blog/retry-web-api-calls-without-paying-twice) goes deeper. | Status | Code | What to do | | --- | --- | --- | | 429 | `rate_limited` | Wait the `Retry-After` seconds, then retry the same page | | 502 | `source_error` | Retry with backoff | | 503 | `temporarily_unavailable` | Wait the `Retry-After` seconds, then retry | | 402 | `insufficient_credits` | Stop. Nothing more is debited and there's no automatic top-up | | 401 | `invalid_api_key` | Stop. Fix the key | | 422 | `invalid_parameters` | Stop. Fix the request, for example a malformed or missing parameter | The `get_page` function above retries the first three, and on network errors too. It backs off 1, 2, then 4 seconds, or waits longer when a `Retry-After` header asks for it, the standard response to [429 Too Many Requests](https://www.rfc-editor.org/rfc/rfc6585#section-4). Everything else falls through to `collect`, which records the reason and returns the items it already has. For long pulls, save `seen` to disk every few pages, and cache pages you've already fetched so a rerun doesn't pay for them again (see [caching API responses](https://perceptlayer.com/blog/cache-web-data-api-responses)). Then a 402 or a crashed process costs you the remaining pages, not the whole run. Resume by starting from the last saved page number and loading the saved `product_id` values back into `seen`. For more, see the [errors docs](https://docs.perceptlayer.com/errors) and the [rate limit docs](https://docs.perceptlayer.com/rate-limits). If you're calling the API through the `requests` library as above, the [Requests documentation](https://requests.readthedocs.io/en/latest/) covers sessions, which reuse connections across a pull of hundreds of pages. Get a free API key at [/signup](https://perceptlayer.com/signup), set `API_KEY`, and run `collect` with a `credit_ceiling` of 20 to see the whole loop end to end for 20 credits or fewer. ## APIs in this post - [eBay **1 credit per call** · 3 endpoints](https://perceptlayer.com/apis/ebay) ## Frequently asked questions ### How many pages should I fetch per query? Fetch until `has_more` is false or your cap is hit, whichever comes first. The docs don't state a maximum page depth, so don't hardcode one. Set `max_pages` and `credit_ceiling` from what you're willing to spend, then let the loop stop itself. Each page is 1 credit. ### Does a retried page get billed twice? Not for a 429, 502 or 503. Only a successful response debits credits, and a failed call returns `X-Credits-Used: 0`. A client timeout is different: there's no idempotency key, so a request whose reply was lost may have succeeded and been billed, and the retry then costs another credit. If you re-request a page you already have, that's a new successful call and costs 1 credit; [caching responses](https://perceptlayer.com/blog/cache-web-data-api-responses) avoids that. ### Should I use per\_page 60, 120 or 240? Use 240 when you want the most results per credit, because the endpoint costs 1 credit per successful call, so fewer, larger pages mean fewer calls. Use 60 when you only need the top results and want smaller responses. The docs list only those three values, so stick to them. ### What happens when my credits run out mid-pull? The next call returns `insufficient_credits` (402) and nothing more is debited. The service is prepaid with no automatic top-up. Your loop should stop on that code and keep the results it has. The `X-Credits-Remaining` header on each success shows the balance after that call, so you can stop early, as the loop above does. ### Can I paginate through an AI agent instead of a script? Yes. The MCP server exposes `call_endpoint`, which spends the endpoint's credits on success only and reports credits used and remaining. An agent can request one page at a time and stop on `has_more`. Set a page cap in its instructions. See the [MCP docs](https://docs.perceptlayer.com/ai-agents). pagination python ebay credits data engineering ## More from the blog 1. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 12 Oct 2026 Perceptlayer 7 min read ## [Incremental API Data Pulls: Fetch Only New Records](https://perceptlayer.com/blog/incremental-api-data-pulls) Pull only new listings: sort newest first, keep a high-water mark or a seen-ID set per query, and stop paging early. Python tested on Craigslist and eBay data. 2. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 29 Sept 2026 Perceptlayer 6 min read ## [Save API JSON to Parquet in Python](https://perceptlayer.com/blog/save-api-json-to-parquet-python) Write API JSON to date-partitioned Parquet with pyarrow and a fixed schema, so a price that arrives as 441 one day and 393.73 the next doesn't break it. 3. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 14 Aug 2026 Perceptlayer 7 min read ## [Normalise Prices Across Currencies and Sources in Python](https://perceptlayer.com/blog/normalize-prices-multiple-currencies) Parse '162,90 €' and $89.99 into Decimals, tag each price with its currency, convert with a dated ECB reference rate, and keep the original value for audit.