Blog / Data engineering

Deduplicate Listings Across Sources: Matching Without IDs

28 Jul 2026Perceptlayer10 min read

Two sources never share a listing ID, so you match on what the car is: normalised make and model, first-registration year, price and mileage. Group candidates by make and model, score each pair on the remaining fields, and accept only pairs that clear a threshold. This tutorial builds that matcher in Python against the AutoScout24 and AutoTrader UK search endpoints, and states where it will get things wrong.

Why can't you deduplicate listings by ID?

Each marketplace issues its own identifiers. AutoScout24 returns a listing_id UUID, and AutoTrader UK returns an advert_id; neither means anything on the other site. A dealer who advertises one car in both places creates two records with no key in common, so a join on ID finds nothing and you have to compare attributes instead.

That's record linkage, a problem statisticians formalised long before web data existed (Fellegi and Sunter, A Theory for Record Linkage, Journal of the American Statistical Association, 1969; see also the overview of record linkage). The practical recipe hasn't changed: normalise the fields, generate candidate pairs, score them, and decide.

Two caveats before the code. The two sources may overlap less than you'd expect, since AutoTrader UK covers the UK and AutoScout24 takes a market parameter for other countries, so check how many matches your own filters produce. The same matcher works unchanged for two feeds of the same market, and for duplicates inside one source (a dealer relisting a car). And the matcher gives you probabilities of sameness, not proof.

Which fields do the two sources share?

Both search endpoints return make, model, year and price, but under different names and units. AutoScout24 gives the model group, first-registration year, price and mileage in kilometres. AutoTrader's recorded search response carries make, model, year and price inside tracking_context, and it has no mileage field. That gap shapes the whole design.

Here's the shape of the AutoScout24 fields we use, trimmed from the recorded response:

{
  "listing_id": "f81b29b2-19e3-4418-8801-03578ed51631",
  "price": { "amount": 5700, "currency": "EUR", … },
  "mileage": { "value_km": 215000, … },
  "first_registration": { "raw": "06/2013", "year": 2013, "month": 6 },
  "make": "BMW",
  "model": "316",
  "model_group": "3er",
  …
}

And the AutoTrader UK fields, also trimmed from its recorded response:

{
  "advert_id": "202606063062554",
  "title": "BMW 5 Series",
  "price": { "amount": 2200, "currency": "GBP", "display": "£2,200" },
  …
  "tracking_context": { "price": 2200, "make": "BMW", "model": "5 Series", "year": 2014, … }
}

I haven't got a recorded AutoTrader listing-detail response, so I won't claim that the listing endpoint returns mileage. If you find it does, fetch it only for candidate pairs and feed it into the same scorer; the code below already handles a mileage that is present on one side only.

How do you fetch both sources?

Call each search endpoint with the filters you care about and read the listings array. Each page costs 1 credit on either endpoint, so five pages from each source is 5 × 2 × 1 = 10 credits per run. Send the key in the X-API-Key header, as described in authentication.

The request for AutoScout24 uses market, make_id, model_id, sort, page and limit; the request for AutoTrader uses postcode, make, price_search_type, sort and page. Both come from the endpoint parameter lists on the AutoScout24 search page and the AutoTrader UK search page.

import os, re, time
from collections import defaultdict
from dataclasses import dataclass
from itertools import combinations
from typing import Optional
import requests

BASE = "https://api.perceptlayer.com"
HEADERS = {"X-API-Key": os.environ["API_KEY"]}


def get(path, params):
    for _ in range(3):
        r = requests.get(BASE + path, params=params, headers=HEADERS, timeout=30)
        if r.status_code in (429, 503):
            time.sleep(int(r.headers.get("Retry-After", "5")))
            continue
        r.raise_for_status()
        return r.json()["data"]
    raise RuntimeError(f"gave up on {path}")

The get helper waits for the Retry-After seconds on a 429 or 503, which the rate limits page describes. It deliberately doesn't retry timeouts: a request whose reply was lost may have succeeded and been billed, a case covered in retrying web API calls without paying twice and handling errors in Python.

Step 1: Normalise each listing into one record

Convert both shapes into a single Car record before comparing anything. Every later step then reads one schema, and every source-specific quirk lives in one small function per source. Prices stay in their original currency, with a property that converts to euros using a rate you supply.

TO_EUR = {"EUR": 1.0, "GBP": float(os.environ["GBP_TO_EUR"])}


@dataclass
class Car:
    uid: str
    make: str
    model: str
    year: int
    price: float
    currency: str
    km: Optional[float] = None

    @property
    def eur(self):
        return self.price * TO_EUR[self.currency]


def from_autoscout24(l):
    return Car(
        uid="as24:" + l["listing_id"],
        make=l["make"],
        model=l.get("model_group") or l["model"],
        year=l["first_registration"]["year"],
        price=l["price"]["amount"],
        currency=l["price"]["currency"],
        km=l["mileage"]["value_km"],
    )


def from_autotrader(l):
    t = l["tracking_context"]
    return Car(
        uid="at:" + l["advert_id"],
        make=t["make"],
        model=t["model"],
        year=t["year"],
        price=l["price"]["amount"],
        currency=l["price"]["currency"],
    )

Set GBP_TO_EUR from a rate source you trust on the day you run. Don't hard-code one: a stale rate quietly widens every price gap. Add a source by writing one more from_… function and, if it uses another currency, one more entry in TO_EUR.

Next come the fetchers. They stop when pagination.has_more is false, which both recorded responses include. For longer loops with safe stopping rules, see paginating a web data API in Python.

def fetch_autoscout24(pages=5):
    cars = []
    for page in range(1, pages + 1):
        data = get("/v1/autoscout24/search", {
            "market": "de", "make_id": "13", "model_id": "1639",
            "sort": "price", "page": page, "limit": 20,
        })
        cars += [from_autoscout24(l) for l in data["listings"]]
        if not data["pagination"]["has_more"]:
            break
    return cars


def fetch_autotrader(pages=5):
    cars = []
    for page in range(1, pages + 1):
        data = get("/v1/autotrader-uk/search", {
            "postcode": "SW1A1AA", "price_search_type": "total",
            "make": "BMW", "sort": "relevance", "page": page,
        })
        cars += [from_autotrader(l) for l in data["listings"]]
        if not data["pagination"]["has_more"]:
            break
    return cars

Step 2: How do you normalise make and model names?

Compare names after stripping case, spaces and punctuation, then map known spelling differences to one form. This is the hand-maintained part of any matcher, because marketplaces disagree on names. The recorded AutoScout24 listing has model_group set to 3er, while AutoTrader's recorded model for a 5-series car is 5 Series.

MAKE_ALIASES = {"vw": "volkswagen", "mercedes": "mercedesbenz"}


def make_key(text):
    t = re.sub(r"[^a-z0-9]", "", text.lower())
    return MAKE_ALIASES.get(t, t)


def model_key(text):
    t = re.sub(r"[^a-z0-9]", "", text.lower())
    return re.sub(r"^(\d)(er|series)$", r"\1series", t)

Here model_key("3er") and model_key("3 Series") both return 3series, and model_key("X5") returns x5. Those three outputs are from running this code. Only 3er and 5 Series come from recorded responses; 3 Series is my test input, written in the same style as the recorded 5 Series, and I haven't recorded AutoTrader returning it. Check the spelling against your own data.

Keep the alias tables short and grow them from review output. The AutoScout24 taxonomy endpoint takes include[] values makes and models, and the AutoTrader UK taxonomy endpoint takes postcode and an optional make. Compare what each returns for your make to find the spellings that disagree.

Step 3: How do you cut down the pairs to compare?

Group listings into blocks that share the normalised make and model, and only compare pairs inside a block. A block is the cheap filter that makes fuzzy scoring affordable. Without it, N listings produce N(N-1)/2 comparisons, and most of them pit a Golf against an X5.

def compare(cars):
    blocks = defaultdict(list)
    for c in cars:
        blocks[(make_key(c.make), model_key(c.model))].append(c)
    for group in blocks.values():
        for a, b in combinations(group, 2):
            s, parts = score(a, b)
            yield a, b, round(s, 3), parts, label(s, parts)

Blocking on make and model assumes both sources spell them well enough to land in the same block. A pair whose model names normalise differently is never compared, and that's a missed match you won't see. Review the biggest singleton blocks now and then to catch spellings your alias tables don't cover yet.

Step 4: How do you score a candidate pair?

Score each field from 0 to 1, weight the scores, and average over the fields both records have. Year, price and mileage each get a closeness score, and mileage counts most because two different cars rarely share an odometer reading. Tolerances absorb the noise: asking prices drift, and exchange rates move.

PRICE_TOL = 0.10
KM_TOL, KM_FLOOR = 0.03, 1000
WEIGHTS = {"year": 1, "price": 2, "mileage": 3}
MATCH, REVIEW = 0.85, 0.80


def closeness(a, b, rel, floor=0):
    width = max(max(a, b) * rel, floor)
    return max(0.0, 1 - abs(a - b) / width) if width else 1.0


def score(a, b):
    """Score two cars from the same block: 0 (different) to 1 (same)."""
    if abs(a.year - b.year) > 1:
        return 0.0, {}
    parts = {
        "year": 1.0 if a.year == b.year else 0.5,
        "price": closeness(a.eur, b.eur, PRICE_TOL),
    }
    if a.km is not None and b.km is not None:
        parts["mileage"] = closeness(a.km, b.km, KM_TOL, KM_FLOOR)
    total = sum(WEIGHTS[k] * v for k, v in parts.items())
    return total / sum(WEIGHTS[k] for k in parts), parts


def label(s, parts):
    if "mileage" not in parts:
        return "review" if s >= REVIEW else "different"
    if s >= MATCH:
        return "match"
    return "review" if s >= REVIEW else "different"

A few choices deserve a sentence each:

• Year allows a gap of one. Sources may define the year differently (registration versus manufacture), so a one-year gap scores 0.5 rather than failing outright. Two years apart is rejected.
• Price tolerance is 10% of the larger price, after conversion. It covers the exchange rate and asking-price drift. Tighten it if you match within one currency.
• Mileage tolerance is 3%, with a floor of 1,000 km. The floor stops a small odometer difference from zeroing the score on a low-mileage car.
• No mileage means no automatic match. Without the heaviest signal, year and price alone can clear the bar for two different cars, so label caps those pairs at review.

These numbers are starting points I chose, not measured optima. Tune them on pairs you've labelled yourself.

Step 5: How do you group matches into clusters?

Union the pairs labelled match, and treat each connected set as one physical car. A union-find structure does this in near-linear time (disjoint-set data structure). It also merges chains, so A matching B and B matching C puts all three together even if A and C would not match directly.

def clusters(cars, pairs):
    parent = {c.uid: c.uid for c in cars}

    def find(x):
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x

    for a, b, _, _, lab in pairs:
        if lab == "match":
            parent[find(a.uid)] = find(b.uid)
    groups = defaultdict(list)
    for c in cars:
        groups[find(c.uid)].append(c.uid)
    return list(groups.values())


if __name__ == "__main__":
    cars = fetch_autoscout24() + fetch_autotrader()
    pairs = list(compare(cars))
    for a, b, s, parts, lab in pairs:
        if lab != "different":
            print(lab, s, a.uid, b.uid, parts)
    print(clusters(cars, pairs))

Send review pairs to a person or a second check rather than into the clusters. They're the ones where an extra signal, such as comparing photos or calling the listing endpoint for more fields, earns its credit.

What does the matcher do on real listings?

On the two recorded AutoScout24 listings it correctly refuses to merge a near-miss. Both are BMW 3-series diesels at about €5,700 to €5,800, so a price-only matcher would fold them together. Their mileage separates them: 215,000 km against 177,000 km. I ran the code above on the recorded fields to see what it does.

The recorded AutoTrader listings (BMW 5 Series, 2014, £2,200, and BMW X5, 2005, £2,700) land in different blocks from the AutoScout24 pair and from each other, so nothing is compared. That is the right outcome, but it doesn't show a true match, and the recordings contain none.

To exercise the match path I built a synthetic test pair, which is my input and not API output: a 3er from 2013 at €5,700 and 215,000 km, and a 3 Series from 2013 at £4,900 and 215,000 km, with GBP_TO_EUR=1.15 as a test value, not a quoted rate. The scorer returns 0.962 and labels it match. Take away the second car's mileage and the score is 0.923, labelled review. The clusters function then returns the first two as one group, with the mileage-less third car left alone.

What are the false-match trade-offs?

Every threshold trades two errors against each other. Raise MATCH and you leave true duplicates apart, which inflates counts and averages. Lower it and you merge two different cars, which deletes a real listing from your data. Which error costs more depends on what you do with the output.

Some errors are structural, whatever you tune:

• Identical twins. A dealer with three same-year, same-spec cars at similar prices and mileage will look like one car. Group by seller or location as an extra guard if your data has them.
• Chained merges. Union-find propagates one bad link into a whole cluster. Cap cluster size and inspect anything larger than the number of sources.
• Price changes between fetches. A dealer who cuts the price after your first call pushes a true pair apart. The wider the time gap between fetches, the more you rely on mileage.
• Missing fields. Any source without mileage leaves you with year and price, which is why those pairs stop at review.

Measure instead of guessing. Label a few hundred pairs by hand, including the review band, then count how many confirmed matches fall below MATCH and how many rejected pairs sit above it. The data quality checks in web data quality checks show where to put that measurement in a pipeline.

What does a run cost?

A run costs 1 credit per search call, one call per page. Five pages from each source is 10 credits. Running that daily for 30 days is 30 × 10 = 300 credits, inside the 1,000 free credits every account gets each month after verifying an email address. See credits for how billing works and pricing for what credits cost.

Failed calls cost nothing and return X-Credits-Used: 0. If you'd rather not refetch pages you've already seen, caching API responses cuts credits further. Rate limits are per account and counted per minute; if you fetch both sources at once, see concurrent requests under rate limits.

Run the fetchers with your own filters next. Get a free API key at sign-up, try a search in the playground on the AutoScout24 API page or the AutoTrader UK API page, then paste the output into compare. For the fields these sources expose, read used car listing data for price analysis, and for loading the result, loading API JSON into Postgres.

deduplicationrecord linkageused carspythondata quality