--- title: "Incremental API Data Pulls: Fetch Only New Records" description: "Pull only new listings: sort newest first, keep a high-water mark or a seen-ID set per query, and stop paging early. Python tested on Craigslist and eBay data." url: https://perceptlayer.com/blog/incremental-api-data-pulls --- [Blog](https://perceptlayer.com/blog) / [Data engineering](https://perceptlayer.com/blog/category/data-engineering) # Incremental API Data Pulls: Fetch Only New Records 12 Oct 2026 Perceptlayer 7 min read An incremental data load from an API fetches only the records that changed since the last run. For listing data, that means three things: sort newest first, remember where the last run stopped (a timestamp or a set of IDs, per query), and stop paging as soon as you reach records you already have. On Craigslist the mark is a timestamp; eBay results carry no date, so the mark is a set of item IDs. ## Key takeaways - Keep one mark per query: the newest `bumped_at` for Craigslist, the set of seen `product_id` values for eBay, stored between runs. - Stop paging at the first record older than the mark, after re-reading a small overlap window to absorb clock skew and slightly out-of-order results. - A Craigslist posting can come back with its old `posting_id` and a new `bumped_at` when its owner renews it, so upsert by ID rather than appending. - eBay's recorded search results were all `sponsored: true`; sponsored slots may not follow the sort order, so only organic results should decide when to stop. - The recorded Craigslist search reports 821 results at 360 a page, 3 calls for a full pull; an incremental run usually needs 1, at 1 credit per call. ## Why pull incrementally at all? Because full pulls repeat work you've paid for. A search that returns 821 Craigslist postings costs the same three pages every run, even if only five postings are new. Incremental pulls cut that to the pages that hold new records, usually one, and they make each run's output a list of changes, which is what downstream jobs want anyway. The pattern is the same for any listing source: a sort that puts the newest records first, a field that says when a record is new, and state that survives between runs. What changes between sources is which field marks "new". The two sources in this guide show both cases: Craigslist gives you a timestamp, eBay doesn't. ## What do the two sources give you to work with? Each search response has a list of results and a flag that says another page exists. What differs is the field you can track. The excerpts below are from recorded responses, trimmed with `…`. The [Craigslist Search endpoint](https://perceptlayer.com/apis/craigslist/search) returned this for bikes in Portland: ```json { "success": true, "data": { "total": 821, "page": 1, "per_page": 360, "has_more": true, "results": [ { "position": 1, "id": "wnD16b4hEY7DaHprfGYGAg", "posting_id": 7988730852, "title": "21\" Specialized Sirrus A1 commuter", "price": 250, "currency": "USD", "location": { "site": "portland", "subarea": "mlt", … }, "bumped_at": "2026-10-23T20:30:33Z", … }, … ] } } ``` The [eBay Search endpoint](https://perceptlayer.com/apis/ebay/search) returned this for `q=coffee maker`, with no date on any result: ```json { "success": true, "data": { "query": "coffee maker", "page": 1, "per_page": 60, "has_more": true, "results": [ { "product_id": "366702966104", "title": "Mr. Coffee 4-in-1 Single-Serve Latte, Iced, and Hot Coffee", "price": { "raw": "$60.00", "extracted": 60, "currency": "USD" }, "sponsored": true, … }, … ], "total_results": 110000, … } } ``` | | Craigslist search | eBay search | | --- | --- | --- | | Newest-first sort | `sort=date` | `sort` takes `1`, `7`, `10`, `12`, `15`, `16` | | Field that marks "new" | `bumped_at` | none; use `product_id` | | Stable ID | `posting_id` (also `id`) | `product_id` | | Page size | `per_page` 360 in the recording | `per_page`: `60`, `120` or `240` | | More pages | `has_more` | `has_more` | | Mark to store | Newest `bumped_at` per query | Seen `product_id` set per query | On eBay's own site, the `_sop=10` URL parameter sorts by newly listed and `_sop=12` by best match, according to eBay's community forum ( [eBay Community](https://community.ebay.com/t5/Report-eBay-Technical-Issues/Best-match-newly-listed-ending-soonest-drop-down-NOT-WORKING/m-p/30712217/highlight/true), 2026). The endpoint's `sort` values look like the same codes, and the recorded related-search links carry `_sop=12`, but we haven't confirmed that `sort=10` means newly listed here; check it in the playground before relying on it. ## How do you pull Craigslist incrementally? Sort by date, read pages until you reach a posting bumped before the last run's mark, and save the newest `bumped_at` you saw as the next mark. Re-read a short overlap before the mark, 30 minutes in the code below, so a posting whose bump time lands a little out of order isn't lost. These are the requests the code sends: ```bash curl "https://api.perceptlayer.com/v1/craigslist/search?site=portland&category=bia&query=bike&min_price=50&max_price=300&sort=date&page=1" \ -H "X-API-Key: $API_KEY" ``` The overlap matters because the order isn't guaranteed to be strict. The recorded request didn't set `sort`, and its third posting, bumped at 20:28:59, comes after one bumped at 20:28:35. With `sort=date` the order should hold, but a few minutes of overlap is cheap insurance. The cost is a few re-read postings each run, which the upsert step absorbs. ```python """Incremental pulls: fetch only listings that are new or bumped since the last run.""" import json import os import pathlib from datetime import datetime, timedelta import requests API = "https://api.perceptlayer.com/v1" HEADERS = {"X-API-Key": os.environ["API_KEY"]} STATE = pathlib.Path("state.json") OVERLAP = timedelta(minutes=30) # re-read a little before the mark, in case of clock skew MAX_PAGES = 5 spent = 0 def get(path, **params): global spent r = requests.get(f"{API}/{path}", headers=HEADERS, params=params, timeout=60) spent += int(r.headers.get("X-Credits-Used", "0")) r.raise_for_status() return r.json()["data"] def ts(s): return datetime.fromisoformat(s.replace("Z", "+00:00")) def craigslist_since(params, state): """Newest first; stop at the first posting bumped before the high-water mark.""" key = "craigslist:" + json.dumps(params, sort_keys=True) mark = ts(state[key]) - OVERLAP if key in state else None fresh, newest = [], None for page in range(1, MAX_PAGES + 1): data = get("craigslist/search", **params, sort="date", page=page) reached_mark = False for row in data["results"]: bumped = ts(row["bumped_at"]) newest = max(newest or bumped, bumped) if mark and bumped < mark: reached_mark = True break fresh.append(row) if reached_mark or not data.get("has_more"): break if newest: state[key] = newest.isoformat() return fresh ``` The mark is stored per query, keyed by the query's parameters, so a change in price band or category starts a fresh history. `MAX_PAGES` caps a first run or a run after a long gap, so the job can't page through a whole city at once. ## How do you handle bumps and reposts? Upsert, don't append. Craigslist lets owners of free postings move them back to the top every 48 to 72 hours by renewing them ( [Craigslist help: renew](https://www.craigslist.org/about/help/posting/modify/renew), 2026). A renewed posting comes back with its old `posting_id` and a new `bumped_at`, so it reappears in an incremental pull. The recording shows one: `posting_id` 7939313997, far older than its neighbours' 7988… IDs, bumped the same evening. Key your table on `posting_id` and update the price and `bumped_at` when one comes back. That turns a bump into a useful event: a renewed posting with a lower price is a price cut. A repost is different: Craigslist says a repost is a new posting, while the original stays up under its original date ( [Craigslist help: repost](https://www.craigslist.org/about/help/repost), 2026). Reposts get new IDs, so catch them by matching title, price and location, as in [deduplicating listings across sources](https://perceptlayer.com/blog/deduplicate-listings-across-sources). ## How do you pull eBay when results have no date? Track IDs instead of time. Keep the set of `product_id` values you've seen for each query, sort newest first, and stop at the first page that has no unseen organic item. The rule needs the word "organic": every result in the recorded eBay response is `sponsored: true`, and sponsored slots may not follow the requested sort, so they can't tell you where new items end. ```python def ebay_since(params, state, sort=10): """No date in the results: stop at the first page with no unseen organic item.""" key = "ebay:" + json.dumps(params, sort_keys=True) seen = set(state.get(key, [])) fresh = [] for page in range(1, MAX_PAGES + 1): data = get("ebay/search", **params, sort=sort, page=page, per_page=60) new = [r for r in data["results"] if r["product_id"] not in seen] organic = [r for r in data["results"] if not r.get("sponsored")] organic_new = [r for r in new if not r.get("sponsored")] fresh.extend(new) seen.update(r["product_id"] for r in data["results"]) # Sponsored slots may not follow the sort order, so organic results decide when to stop. if not new or (organic and not organic_new) or not data.get("has_more"): break state[key] = sorted(seen) return fresh if __name__ == "__main__": state = json.loads(STATE.read_text()) if STATE.exists() else {} bikes = craigslist_since({"site": "portland", "category": "bia", "query": "bike", "min_price": 50, "max_price": 300}, state) coffee = ebay_since({"q": "coffee maker"}, state) STATE.write_text(json.dumps(state)) print(f"{len(bikes)} new or bumped postings, {len(coffee)} new eBay items, {spent} credits") ``` A broad eBay query like this one reports 110,000 results, so a full pull is out of reach anyway; the incremental pull is the only practical way to follow it. Prune the seen set over time, for example by dropping IDs not seen for 30 days, or it grows without limit. For general page-loop safety, see [paginating a web data API in Python](https://perceptlayer.com/blog/paginate-web-data-api-python). ## What happened when we ran it on recorded data? We ran both functions twice with `requests.get` mocked: page 1 returned the recorded response, page 2 an empty last page. The recordings are short samples, 3 results each, so this checks the stopping logic, not real volumes. We haven't run the code against the live API. | Run | Craigslist, 30-minute overlap | Craigslist, no overlap | eBay | | --- | --- | --- | --- | | First | 3 postings, 2 calls | 3 postings, 2 calls | 3 items, 2 calls | | Second, same data | 3 postings, 2 calls | 1 posting, 1 call | 0 items, 1 call | The eBay pull did what it should: nothing new, one call. The Craigslist overlap shows its cost: all three recorded postings were bumped within 30 minutes of each other, so the second run re-read them all. With no overlap, the second run stopped at the posting that set the mark. In production, runs are hours apart and only postings in the last 30 minutes of the window repeat; the upsert by `posting_id` makes those repeats harmless. ## How many credits does an incremental pull save? Each call costs 1 credit on success, and failed calls cost nothing and return `X-Credits-Used: 0`, so savings are pages not fetched. The recorded Craigslist search reports 821 results at 360 a page: a full pull is 3 calls. An incremental run that finds its mark on page 1 is 1 call, saving 2 credits per query per run. The [credits docs](https://docs.perceptlayer.com/credits) explain billing. | Job (Craigslist, 20 queries like the recorded one) | Calls a day | Credits a month | | --- | --- | --- | | Full pull, hourly | 20 × 3 × 24 = 1,440 | 43,200 | | Incremental, hourly, 1 page a run | 20 × 1 × 24 = 480 | 14,400 | | Incremental, every 4 hours | 20 × 1 × 6 = 120 | 3,600 | Worked example: 20 queries hourly drops from 43,200 credits a month to 14,400 when each run needs one page, a saving of 28,800 credits. Busy queries will sometimes need two pages; budget for that. A Starter pack of 10,000 credits covers the four-hourly job for almost three months, and purchased credits last 12 months. See [pricing](https://perceptlayer.com/pricing) for what credits cost. Hourly runs of 20 queries stay well inside the free tier's 1 request a second; the [rate limits docs](https://docs.perceptlayer.com/rate-limits) cover paid rates. For when to run the job, see [scheduling web data pulls](https://perceptlayer.com/blog/schedule-web-data-pulls-cron-queue), and for loading the rows, [loading API JSON into Postgres](https://perceptlayer.com/blog/load-web-data-json-into-postgres), which covers upserts. To try the searches, [get a free API key](https://perceptlayer.com/signup) and run them in the playground on the [Craigslist API page](https://perceptlayer.com/apis/craigslist) and the [eBay API page](https://perceptlayer.com/apis/ebay). Data is collected from publicly available sources and returned as found at the time of the call. ## APIs in this post - [Craigslist **1 credit per call** · 4 endpoints](https://perceptlayer.com/apis/craigslist) - [eBay **1 credit per call** · 3 endpoints](https://perceptlayer.com/apis/ebay) ## Frequently asked questions ### What is an incremental data load from an API? It's a pull that fetches only records that are new or changed since the previous run. You sort results newest first, store a mark per query (a timestamp such as Craigslist's `bumped_at`, or a set of seen IDs where results carry no date, as on eBay), and stop paging when you reach records older than the mark. ### What if the API has no updated-at field? Track IDs. Keep the set of record IDs you've seen per query, request a newest-first sort and stop at the first page with no unseen items. On eBay that means `product_id`, ignoring `sponsored` results when you decide to stop, since sponsored slots may not follow the sort order. Prune old IDs so the set doesn't grow forever. ### How do I avoid missing records between runs? Re-read an overlap window before the mark, such as 30 minutes, and upsert by ID so the repeats don't create duplicates. Don't stop on the first older record unless results are sorted by date: the recorded Craigslist search, made without `sort`, returned one posting out of time order. Cap pages per run so a long gap can't run away. ### How much do incremental pulls save? The pages you don't fetch. The recorded Craigslist search reports 821 results at 360 a page, 3 credits for a full pull; an incremental run that ends on page 1 costs 1 credit. For 20 queries hourly, that's 14,400 credits a month instead of 43,200. Failed calls cost nothing. ### How should I handle reposted and renewed listings? Renewed Craigslist postings keep their `posting_id` and get a new `bumped_at`, so upsert them and record any price change. Reposts get a new ID while the original stays up, so match them on title, price and location instead. Keep both events: a renewal at a lower price is a price cut worth tracking. incremental load python pagination data engineering craigslist ebay ## More from the blog 1. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 6 Aug 2026 Perceptlayer 8 min read ## [Paginate a Web Data API in Python: Safe Page Loops](https://perceptlayer.com/blog/paginate-web-data-api-python) Page through the eBay search endpoint in Python with has\_more, dedupe by product\_id and a credit ceiling. Each page costs 1 credit. 2. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 29 Sept 2026 Perceptlayer 6 min read ## [Save API JSON to Parquet in Python](https://perceptlayer.com/blog/save-api-json-to-parquet-python) Write API JSON to date-partitioned Parquet with pyarrow and a fixed schema, so a price that arrives as 441 one day and 393.73 the next doesn't break it. 3. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 14 Aug 2026 Perceptlayer 7 min read ## [Normalise Prices Across Currencies and Sources in Python](https://perceptlayer.com/blog/normalize-prices-multiple-currencies) Parse '162,90 €' and $89.99 into Decimals, tag each price with its currency, convert with a dated ECB reference rate, and keep the original value for audit.