Blog / Data engineering

Incremental API Data Pulls: Fetch Only New Records

12 Oct 2026Perceptlayer7 min read

An incremental data load from an API fetches only the records that changed since the last run. For listing data, that means three things: sort newest first, remember where the last run stopped (a timestamp or a set of IDs, per query), and stop paging as soon as you reach records you already have. On Craigslist the mark is a timestamp; eBay results carry no date, so the mark is a set of item IDs.

Why pull incrementally at all?

Because full pulls repeat work you've paid for. A search that returns 821 Craigslist postings costs the same three pages every run, even if only five postings are new. Incremental pulls cut that to the pages that hold new records, usually one, and they make each run's output a list of changes, which is what downstream jobs want anyway.

The pattern is the same for any listing source: a sort that puts the newest records first, a field that says when a record is new, and state that survives between runs. What changes between sources is which field marks "new". The two sources in this guide show both cases: Craigslist gives you a timestamp, eBay doesn't.

What do the two sources give you to work with?

Each search response has a list of results and a flag that says another page exists. What differs is the field you can track. The excerpts below are from recorded responses, trimmed with …. The Craigslist Search endpoint returned this for bikes in Portland:

{
  "success": true,
  "data": {
    "total": 821,
    "page": 1,
    "per_page": 360,
    "has_more": true,
    "results": [
      {
        "position": 1,
        "id": "wnD16b4hEY7DaHprfGYGAg",
        "posting_id": 7988730852,
        "title": "21\" Specialized Sirrus A1 commuter",
        "price": 250,
        "currency": "USD",
        "location": { "site": "portland", "subarea": "mlt", … },
        "bumped_at": "2026-10-23T20:30:33Z",
        …
      },
      …
    ]
  }
}

The eBay Search endpoint returned this for q=coffee maker, with no date on any result:

{
  "success": true,
  "data": {
    "query": "coffee maker",
    "page": 1,
    "per_page": 60,
    "has_more": true,
    "results": [
      {
        "product_id": "366702966104",
        "title": "Mr. Coffee 4-in-1 Single-Serve Latte, Iced, and Hot Coffee",
        "price": { "raw": "$60.00", "extracted": 60, "currency": "USD" },
        "sponsored": true,
        …
      },
      …
    ],
    "total_results": 110000,
    …
  }
}

On eBay's own site, the _sop=10 URL parameter sorts by newly listed and _sop=12 by best match, according to eBay's community forum (eBay Community, 2026). The endpoint's sort values look like the same codes, and the recorded related-search links carry _sop=12, but we haven't confirmed that sort=10 means newly listed here; check it in the playground before relying on it.

How do you pull Craigslist incrementally?

Sort by date, read pages until you reach a posting bumped before the last run's mark, and save the newest bumped_at you saw as the next mark. Re-read a short overlap before the mark, 30 minutes in the code below, so a posting whose bump time lands a little out of order isn't lost. These are the requests the code sends:

curl "https://api.perceptlayer.com/v1/craigslist/search?site=portland&category=bia&query=bike&min_price=50&max_price=300&sort=date&page=1" \
  -H "X-API-Key: $API_KEY"

The overlap matters because the order isn't guaranteed to be strict. The recorded request didn't set sort, and its third posting, bumped at 20:28:59, comes after one bumped at 20:28:35. With sort=date the order should hold, but a few minutes of overlap is cheap insurance. The cost is a few re-read postings each run, which the upsert step absorbs.

"""Incremental pulls: fetch only listings that are new or bumped since the last run."""
import json
import os
import pathlib
from datetime import datetime, timedelta

import requests

API = "https://api.perceptlayer.com/v1"
HEADERS = {"X-API-Key": os.environ["API_KEY"]}
STATE = pathlib.Path("state.json")
OVERLAP = timedelta(minutes=30)  # re-read a little before the mark, in case of clock skew
MAX_PAGES = 5
spent = 0


def get(path, **params):
    global spent
    r = requests.get(f"{API}/{path}", headers=HEADERS, params=params, timeout=60)
    spent += int(r.headers.get("X-Credits-Used", "0"))
    r.raise_for_status()
    return r.json()["data"]


def ts(s):
    return datetime.fromisoformat(s.replace("Z", "+00:00"))


def craigslist_since(params, state):
    """Newest first; stop at the first posting bumped before the high-water mark."""
    key = "craigslist:" + json.dumps(params, sort_keys=True)
    mark = ts(state[key]) - OVERLAP if key in state else None
    fresh, newest = [], None
    for page in range(1, MAX_PAGES + 1):
        data = get("craigslist/search", **params, sort="date", page=page)
        reached_mark = False
        for row in data["results"]:
            bumped = ts(row["bumped_at"])
            newest = max(newest or bumped, bumped)
            if mark and bumped < mark:
                reached_mark = True
                break
            fresh.append(row)
        if reached_mark or not data.get("has_more"):
            break
    if newest:
        state[key] = newest.isoformat()
    return fresh

The mark is stored per query, keyed by the query's parameters, so a change in price band or category starts a fresh history. MAX_PAGES caps a first run or a run after a long gap, so the job can't page through a whole city at once.

How do you handle bumps and reposts?

Upsert, don't append. Craigslist lets owners of free postings move them back to the top every 48 to 72 hours by renewing them (Craigslist help: renew, 2026). A renewed posting comes back with its old posting_id and a new bumped_at, so it reappears in an incremental pull. The recording shows one: posting_id 7939313997, far older than its neighbours' 7988… IDs, bumped the same evening.

Key your table on posting_id and update the price and bumped_at when one comes back. That turns a bump into a useful event: a renewed posting with a lower price is a price cut. A repost is different: Craigslist says a repost is a new posting, while the original stays up under its original date (Craigslist help: repost, 2026). Reposts get new IDs, so catch them by matching title, price and location, as in deduplicating listings across sources.

How do you pull eBay when results have no date?

Track IDs instead of time. Keep the set of product_id values you've seen for each query, sort newest first, and stop at the first page that has no unseen organic item. The rule needs the word "organic": every result in the recorded eBay response is sponsored: true, and sponsored slots may not follow the requested sort, so they can't tell you where new items end.

def ebay_since(params, state, sort=10):
    """No date in the results: stop at the first page with no unseen organic item."""
    key = "ebay:" + json.dumps(params, sort_keys=True)
    seen = set(state.get(key, []))
    fresh = []
    for page in range(1, MAX_PAGES + 1):
        data = get("ebay/search", **params, sort=sort, page=page, per_page=60)
        new = [r for r in data["results"] if r["product_id"] not in seen]
        organic = [r for r in data["results"] if not r.get("sponsored")]
        organic_new = [r for r in new if not r.get("sponsored")]
        fresh.extend(new)
        seen.update(r["product_id"] for r in data["results"])
        # Sponsored slots may not follow the sort order, so organic results decide when to stop.
        if not new or (organic and not organic_new) or not data.get("has_more"):
            break
    state[key] = sorted(seen)
    return fresh


if __name__ == "__main__":
    state = json.loads(STATE.read_text()) if STATE.exists() else {}
    bikes = craigslist_since({"site": "portland", "category": "bia", "query": "bike",
                              "min_price": 50, "max_price": 300}, state)
    coffee = ebay_since({"q": "coffee maker"}, state)
    STATE.write_text(json.dumps(state))
    print(f"{len(bikes)} new or bumped postings, {len(coffee)} new eBay items, {spent} credits")

A broad eBay query like this one reports 110,000 results, so a full pull is out of reach anyway; the incremental pull is the only practical way to follow it. Prune the seen set over time, for example by dropping IDs not seen for 30 days, or it grows without limit. For general page-loop safety, see paginating a web data API in Python.

What happened when we ran it on recorded data?

We ran both functions twice with requests.get mocked: page 1 returned the recorded response, page 2 an empty last page. The recordings are short samples, 3 results each, so this checks the stopping logic, not real volumes. We haven't run the code against the live API.

The eBay pull did what it should: nothing new, one call. The Craigslist overlap shows its cost: all three recorded postings were bumped within 30 minutes of each other, so the second run re-read them all. With no overlap, the second run stopped at the posting that set the mark. In production, runs are hours apart and only postings in the last 30 minutes of the window repeat; the upsert by posting_id makes those repeats harmless.

How many credits does an incremental pull save?

Each call costs 1 credit on success, and failed calls cost nothing and return X-Credits-Used: 0, so savings are pages not fetched. The recorded Craigslist search reports 821 results at 360 a page: a full pull is 3 calls. An incremental run that finds its mark on page 1 is 1 call, saving 2 credits per query per run. The credits docs explain billing.

Worked example: 20 queries hourly drops from 43,200 credits a month to 14,400 when each run needs one page, a saving of 28,800 credits. Busy queries will sometimes need two pages; budget for that. A Starter pack of 10,000 credits covers the four-hourly job for almost three months, and purchased credits last 12 months. See pricing for what credits cost.

Hourly runs of 20 queries stay well inside the free tier's 1 request a second; the rate limits docs cover paid rates. For when to run the job, see scheduling web data pulls, and for loading the rows, loading API JSON into Postgres, which covers upserts. To try the searches, get a free API key and run them in the playground on the Craigslist API page and the eBay API page. Data is collected from publicly available sources and returned as found at the time of the call.

incremental loadpythonpaginationdata engineeringcraigslistebay