---
title: "Deduplicate Listings Across Sources: Matching Without IDs"
description: "Match car listings from AutoScout24 and AutoTrader UK with no shared ID: normalise make and model, block, then score year, price and mileage in Python."
url: https://perceptlayer.com/blog/deduplicate-listings-across-sources
---

[Blog](https://perceptlayer.com/blog) / [Data engineering](https://perceptlayer.com/blog/category/data-engineering)

# Deduplicate Listings Across Sources: Matching Without IDs

28 Jul 2026 Perceptlayer 10 min read

Two sources never share a listing ID, so you match on what the car is: normalised make and model, first-registration year, price and mileage. Group candidates by make and model, score each pair on the remaining fields, and accept only pairs that clear a threshold. This tutorial builds that matcher in Python against the AutoScout24 and AutoTrader UK search endpoints, and states where it will get things wrong.

## Key takeaways

-   Block on normalised make and model first, then score year, price and mileage inside each block. Comparing every listing with every other one is both slow and noisy.
-   Mileage is the strongest signal. The matcher weights it 3, price 2 and year 1, and it never auto-accepts a pair when one side has no mileage.
-   Two sources can name the same model differently. AutoScout24's recorded `model_group` is `3er`, and AutoTrader writes `5 Series`, so the matcher needs a normaliser for model names.
-   The pipeline costs 1 credit per search page: 5 pages from each source is 10 credits per run.
-   A threshold is a trade-off. Raise it and you keep duplicates; lower it and you merge two different cars.

## Why can't you deduplicate listings by ID?

Each marketplace issues its own identifiers. AutoScout24 returns a `listing_id` UUID, and AutoTrader UK returns an `advert_id`; neither means anything on the other site. A dealer who advertises one car in both places creates two records with no key in common, so a join on ID finds nothing and you have to compare attributes instead.

That's record linkage, a problem statisticians formalised long before web data existed ( [Fellegi and Sunter, A Theory for Record Linkage, Journal of the American Statistical Association](https://doi.org/10.1080/01621459.1969.10501049), 1969; see also the [overview of record linkage](https://en.wikipedia.org/wiki/Record_linkage)). The practical recipe hasn't changed: normalise the fields, generate candidate pairs, score them, and decide.

Two caveats before the code. The two sources may overlap less than you'd expect, since AutoTrader UK covers the UK and AutoScout24 takes a `market` parameter for other countries, so check how many matches your own filters produce. The same matcher works unchanged for two feeds of the same market, and for duplicates inside one source (a dealer relisting a car). And the matcher gives you probabilities of sameness, not proof.

## Which fields do the two sources share?

Both search endpoints return make, model, year and price, but under different names and units. AutoScout24 gives the model group, first-registration year, price and mileage in kilometres. AutoTrader's recorded search response carries make, model, year and price inside `tracking_context`, and it has no mileage field. That gap shapes the whole design.

| Concept | AutoScout24 `search` | AutoTrader UK `search` |
| --- | --- | --- |
| Source ID | `listing_id` | `advert_id` |
| Make | `make` | `tracking_context.make` |
| Model | `model_group` ( `model` is the narrower `316`) | `tracking_context.model` |
| Year | `first_registration.year` | `tracking_context.year` |
| Price | `price.amount`, `price.currency` ( `EUR` in the recording) | `price.amount`, `price.currency` ( `GBP` in the recording) |
| Mileage | `mileage.value_km` | not in the recorded search response |

Here's the shape of the AutoScout24 fields we use, trimmed from the recorded response:

```json
{
  "listing_id": "f81b29b2-19e3-4418-8801-03578ed51631",
  "price": { "amount": 5700, "currency": "EUR", … },
  "mileage": { "value_km": 215000, … },
  "first_registration": { "raw": "06/2013", "year": 2013, "month": 6 },
  "make": "BMW",
  "model": "316",
  "model_group": "3er",
  …
}
```

And the AutoTrader UK fields, also trimmed from its recorded response:

```json
{
  "advert_id": "202606063062554",
  "title": "BMW 5 Series",
  "price": { "amount": 2200, "currency": "GBP", "display": "£2,200" },
  …
  "tracking_context": { "price": 2200, "make": "BMW", "model": "5 Series", "year": 2014, … }
}
```

I haven't got a recorded AutoTrader listing-detail response, so I won't claim that [the listing endpoint](https://perceptlayer.com/apis/autotrader-uk/listing) returns mileage. If you find it does, fetch it only for candidate pairs and feed it into the same scorer; the code below already handles a mileage that is present on one side only.

## How do you fetch both sources?

Call each search endpoint with the filters you care about and read the `listings` array. Each page costs 1 credit on either endpoint, so five pages from each source is 5 × 2 × 1 = 10 credits per run. Send the key in the `X-API-Key` header, as described in [authentication](https://docs.perceptlayer.com/authentication).

The request for AutoScout24 uses `market`, `make_id`, `model_id`, `sort`, `page` and `limit`; the request for AutoTrader uses `postcode`, `make`, `price_search_type`, `sort` and `page`. Both come from the endpoint parameter lists on the [AutoScout24 search page](https://perceptlayer.com/apis/autoscout24/search) and the [AutoTrader UK search page](https://perceptlayer.com/apis/autotrader-uk/search).

```python
import os, re, time
from collections import defaultdict
from dataclasses import dataclass
from itertools import combinations
from typing import Optional
import requests

BASE = "https://api.perceptlayer.com"
HEADERS = {"X-API-Key": os.environ["API_KEY"]}

def get(path, params):
    for _ in range(3):
        r = requests.get(BASE + path, params=params, headers=HEADERS, timeout=30)
        if r.status_code in (429, 503):
            time.sleep(int(r.headers.get("Retry-After", "5")))
            continue
        r.raise_for_status()
        return r.json()["data"]
    raise RuntimeError(f"gave up on {path}")
```

The `get` helper waits for the `Retry-After` seconds on a 429 or 503, which the [rate limits](https://docs.perceptlayer.com/rate-limits) page describes. It deliberately doesn't retry timeouts: a request whose reply was lost may have succeeded and been billed, a case covered in [retrying web API calls without paying twice](https://perceptlayer.com/blog/retry-web-api-calls-without-paying-twice) and [handling errors in Python](https://perceptlayer.com/blog/handle-web-data-api-errors-python).

## Step 1: Normalise each listing into one record

Convert both shapes into a single `Car` record before comparing anything. Every later step then reads one schema, and every source-specific quirk lives in one small function per source. Prices stay in their original currency, with a property that converts to euros using a rate you supply.

```python
TO_EUR = {"EUR": 1.0, "GBP": float(os.environ["GBP_TO_EUR"])}

@dataclass
class Car:
    uid: str
    make: str
    model: str
    year: int
    price: float
    currency: str
    km: Optional[float] = None

    @property
    def eur(self):
        return self.price * TO_EUR[self.currency]

def from_autoscout24(l):
    return Car(
        uid="as24:" + l["listing_id"],
        make=l["make"],
        model=l.get("model_group") or l["model"],
        year=l["first_registration"]["year"],
        price=l["price"]["amount"],
        currency=l["price"]["currency"],
        km=l["mileage"]["value_km"],
    )

def from_autotrader(l):
    t = l["tracking_context"]
    return Car(
        uid="at:" + l["advert_id"],
        make=t["make"],
        model=t["model"],
        year=t["year"],
        price=l["price"]["amount"],
        currency=l["price"]["currency"],
    )
```

Set `GBP_TO_EUR` from a rate source you trust on the day you run. Don't hard-code one: a stale rate quietly widens every price gap. Add a source by writing one more `from_…` function and, if it uses another currency, one more entry in `TO_EUR`.

Next come the fetchers. They stop when `pagination.has_more` is false, which both recorded responses include. For longer loops with safe stopping rules, see [paginating a web data API in Python](https://perceptlayer.com/blog/paginate-web-data-api-python).

```python
def fetch_autoscout24(pages=5):
    cars = []
    for page in range(1, pages + 1):
        data = get("/v1/autoscout24/search", {
            "market": "de", "make_id": "13", "model_id": "1639",
            "sort": "price", "page": page, "limit": 20,
        })
        cars += [from_autoscout24(l) for l in data["listings"]]
        if not data["pagination"]["has_more"]:
            break
    return cars

def fetch_autotrader(pages=5):
    cars = []
    for page in range(1, pages + 1):
        data = get("/v1/autotrader-uk/search", {
            "postcode": "SW1A1AA", "price_search_type": "total",
            "make": "BMW", "sort": "relevance", "page": page,
        })
        cars += [from_autotrader(l) for l in data["listings"]]
        if not data["pagination"]["has_more"]:
            break
    return cars
```

## Step 2: How do you normalise make and model names?

Compare names after stripping case, spaces and punctuation, then map known spelling differences to one form. This is the hand-maintained part of any matcher, because marketplaces disagree on names. The recorded AutoScout24 listing has `model_group` set to `3er`, while AutoTrader's recorded model for a 5-series car is `5 Series`.

```python
MAKE_ALIASES = {"vw": "volkswagen", "mercedes": "mercedesbenz"}

def make_key(text):
    t = re.sub(r"[^a-z0-9]", "", text.lower())
    return MAKE_ALIASES.get(t, t)

def model_key(text):
    t = re.sub(r"[^a-z0-9]", "", text.lower())
    return re.sub(r"^(\d)(er|series)$", r"\1series", t)
```

Here `model_key("3er")` and `model_key("3 Series")` both return `3series`, and `model_key("X5")` returns `x5`. Those three outputs are from running this code. Only `3er` and `5 Series` come from recorded responses; `3 Series` is my test input, written in the same style as the recorded `5 Series`, and I haven't recorded AutoTrader returning it. Check the spelling against your own data.

Keep the alias tables short and grow them from review output. The [AutoScout24 taxonomy endpoint](https://perceptlayer.com/apis/autoscout24/taxonomy) takes `include[]` values `makes` and `models`, and the [AutoTrader UK taxonomy endpoint](https://perceptlayer.com/apis/autotrader-uk/taxonomy) takes `postcode` and an optional `make`. Compare what each returns for your make to find the spellings that disagree.

## Step 3: How do you cut down the pairs to compare?

Group listings into blocks that share the normalised make and model, and only compare pairs inside a block. A block is the cheap filter that makes fuzzy scoring affordable. Without it, N listings produce N(N-1)/2 comparisons, and most of them pit a Golf against an X5.

```python
def compare(cars):
    blocks = defaultdict(list)
    for c in cars:
        blocks[(make_key(c.make), model_key(c.model))].append(c)
    for group in blocks.values():
        for a, b in combinations(group, 2):
            s, parts = score(a, b)
            yield a, b, round(s, 3), parts, label(s, parts)
```

Blocking on make and model assumes both sources spell them well enough to land in the same block. A pair whose model names normalise differently is never compared, and that's a missed match you won't see. Review the biggest singleton blocks now and then to catch spellings your alias tables don't cover yet.

## Step 4: How do you score a candidate pair?

Score each field from 0 to 1, weight the scores, and average over the fields both records have. Year, price and mileage each get a closeness score, and mileage counts most because two different cars rarely share an odometer reading. Tolerances absorb the noise: asking prices drift, and exchange rates move.

```python
PRICE_TOL = 0.10
KM_TOL, KM_FLOOR = 0.03, 1000
WEIGHTS = {"year": 1, "price": 2, "mileage": 3}
MATCH, REVIEW = 0.85, 0.80

def closeness(a, b, rel, floor=0):
    width = max(max(a, b) * rel, floor)
    return max(0.0, 1 - abs(a - b) / width) if width else 1.0

def score(a, b):
    """Score two cars from the same block: 0 (different) to 1 (same)."""
    if abs(a.year - b.year) > 1:
        return 0.0, {}
    parts = {
        "year": 1.0 if a.year == b.year else 0.5,
        "price": closeness(a.eur, b.eur, PRICE_TOL),
    }
    if a.km is not None and b.km is not None:
        parts["mileage"] = closeness(a.km, b.km, KM_TOL, KM_FLOOR)
    total = sum(WEIGHTS[k] * v for k, v in parts.items())
    return total / sum(WEIGHTS[k] for k in parts), parts

def label(s, parts):
    if "mileage" not in parts:
        return "review" if s >= REVIEW else "different"
    if s >= MATCH:
        return "match"
    return "review" if s >= REVIEW else "different"
```

A few choices deserve a sentence each:

-   **Year allows a gap of one.** Sources may define the year differently (registration versus manufacture), so a one-year gap scores 0.5 rather than failing outright. Two years apart is rejected.
-   **Price tolerance is 10% of the larger price, after conversion.** It covers the exchange rate and asking-price drift. Tighten it if you match within one currency.
-   **Mileage tolerance is 3%, with a floor of 1,000 km.** The floor stops a small odometer difference from zeroing the score on a low-mileage car.
-   **No mileage means no automatic match.** Without the heaviest signal, year and price alone can clear the bar for two different cars, so `label` caps those pairs at `review`.

These numbers are starting points I chose, not measured optima. Tune them on pairs you've labelled yourself.

## Step 5: How do you group matches into clusters?

Union the pairs labelled `match`, and treat each connected set as one physical car. A union-find structure does this in near-linear time ( [disjoint-set data structure](https://en.wikipedia.org/wiki/Disjoint-set_data_structure)). It also merges chains, so A matching B and B matching C puts all three together even if A and C would not match directly.

```python
def clusters(cars, pairs):
    parent = {c.uid: c.uid for c in cars}

    def find(x):
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x

    for a, b, _, _, lab in pairs:
        if lab == "match":
            parent[find(a.uid)] = find(b.uid)
    groups = defaultdict(list)
    for c in cars:
        groups[find(c.uid)].append(c.uid)
    return list(groups.values())

if __name__ == "__main__":
    cars = fetch_autoscout24() + fetch_autotrader()
    pairs = list(compare(cars))
    for a, b, s, parts, lab in pairs:
        if lab != "different":
            print(lab, s, a.uid, b.uid, parts)
    print(clusters(cars, pairs))
```

Send `review` pairs to a person or a second check rather than into the clusters. They're the ones where an extra signal, such as comparing photos or calling the listing endpoint for more fields, earns its credit.

## What does the matcher do on real listings?

On the two recorded AutoScout24 listings it correctly refuses to merge a near-miss. Both are BMW 3-series diesels at about €5,700 to €5,800, so a price-only matcher would fold them together. Their mileage separates them: 215,000 km against 177,000 km. I ran the code above on the recorded fields to see what it does.

| Pair | Year score | Price score | Mileage score | Total | Label |
| --- | --- | --- | --- | --- | --- |
| Recorded 2013 (€5,700, 215,000 km) vs recorded 2012 (€5,800, 177,000 km) | 0.5 | 0.828 | 0.0 | 0.359 | `different` |

The recorded AutoTrader listings ( `BMW 5 Series`, 2014, £2,200, and `BMW X5`, 2005, £2,700) land in different blocks from the AutoScout24 pair and from each other, so nothing is compared. That is the right outcome, but it doesn't show a true match, and the recordings contain none.

To exercise the match path I built a synthetic test pair, which is my input and not API output: a `3er` from 2013 at €5,700 and 215,000 km, and a `3 Series` from 2013 at £4,900 and 215,000 km, with `GBP_TO_EUR=1.15` as a test value, not a quoted rate. The scorer returns 0.962 and labels it `match`. Take away the second car's mileage and the score is 0.923, labelled `review`. The clusters function then returns the first two as one group, with the mileage-less third car left alone.

## What are the false-match trade-offs?

Every threshold trades two errors against each other. Raise `MATCH` and you leave true duplicates apart, which inflates counts and averages. Lower it and you merge two different cars, which deletes a real listing from your data. Which error costs more depends on what you do with the output.

| Use | Cheaper error | Lean towards |
| --- | --- | --- |
| Counting distinct cars on the market | Missed duplicates | A high `MATCH`, more `review` |
| Average asking price per model | Either, in moderation | The default thresholds |
| Alerting on a specific car | Missed duplicates (an extra alert) | A high `MATCH` |
| Showing one row per car to users | Merged different cars | A high `MATCH` and manual review |

Some errors are structural, whatever you tune:

-   **Identical twins.** A dealer with three same-year, same-spec cars at similar prices and mileage will look like one car. Group by seller or location as an extra guard if your data has them.
-   **Chained merges.** Union-find propagates one bad link into a whole cluster. Cap cluster size and inspect anything larger than the number of sources.
-   **Price changes between fetches.** A dealer who cuts the price after your first call pushes a true pair apart. The wider the time gap between fetches, the more you rely on mileage.
-   **Missing fields.** Any source without mileage leaves you with year and price, which is why those pairs stop at `review`.

Measure instead of guessing. Label a few hundred pairs by hand, including the `review` band, then count how many confirmed matches fall below `MATCH` and how many rejected pairs sit above it. The data quality checks in [web data quality checks](https://perceptlayer.com/blog/web-data-quality-monitoring-checklist) show where to put that measurement in a pipeline.

## What does a run cost?

A run costs 1 credit per search call, one call per page. Five pages from each source is 10 credits. Running that daily for 30 days is 30 × 10 = 300 credits, inside the 1,000 free credits every account gets each month after verifying an email address. See [credits](https://docs.perceptlayer.com/credits) for how billing works and [pricing](https://perceptlayer.com/pricing) for what credits cost.

Failed calls cost nothing and return `X-Credits-Used: 0`. If you'd rather not refetch pages you've already seen, [caching API responses](https://perceptlayer.com/blog/cache-web-data-api-responses) cuts credits further. Rate limits are per account and counted per minute; if you fetch both sources at once, see [concurrent requests under rate limits](https://perceptlayer.com/blog/concurrent-api-requests-python-rate-limit).

Run the fetchers with your own filters next. Get a free API key at [sign-up](https://perceptlayer.com/signup), try a search in the playground on the [AutoScout24 API page](https://perceptlayer.com/apis/autoscout24) or the [AutoTrader UK API page](https://perceptlayer.com/apis/autotrader-uk), then paste the output into `compare`. For the fields these sources expose, read [used car listing data for price analysis](https://perceptlayer.com/blog/used-car-listing-data-price-analysis), and for loading the result, [loading API JSON into Postgres](https://perceptlayer.com/blog/load-web-data-json-into-postgres).

## APIs in this post

-   [AutoScout24 **1 credit per call** · 6 endpoints](https://perceptlayer.com/apis/autoscout24)
-   [AutoTrader UK **1 credit per call** · 6 endpoints](https://perceptlayer.com/apis/autotrader-uk)

## Frequently asked questions

### Can I deduplicate listings without a shared ID?

Yes. Normalise make, model and year, group listings that share them, then score price and mileage within each group. Accept pairs above a threshold and send borderline pairs to review. Mileage carries the most weight in this tutorial's matcher, so a source without it can only produce `review` pairs, never automatic matches.

### How much does it cost to fetch both sources?

A successful search call costs 1 credit on both the AutoScout24 and AutoTrader UK endpoints. Five pages from each is 5 × 2 × 1 = 10 credits per run. Failed calls cost nothing. Every account gets 1,000 free credits each month after verifying an email address, and [pricing](https://perceptlayer.com/pricing) lists what credits cost.

### Why not match on titles with fuzzy string similarity?

Titles mix trim, engine and marketing text that differ by site: one recorded title is `BMW 316 Touring 316d`, another is `BMW 5 Series`. String distance scores those as unrelated or as similar for the wrong reasons. Structured fields give a more reliable signal. Titles are useful as a tie-breaker inside a block, not as the key.

### Does this work for sources other than cars?

The method does. Normalise the identifying fields, block on a cheap key, and score the numeric fields with tolerances. For property listings you might block on postcode and bedrooms and score price and floor area. You choose the fields and weights from your own data; the car-specific parts here are the make and model normalisers.

### How do I handle currency differences?

Convert to a single currency with a rate you supply, then compare within a tolerance wide enough to cover rate movement and asking-price drift. The matcher keeps the original price and currency on each `Car` and converts at comparison time, so you can re-run with a new rate without refetching.

deduplication record linkage used cars python data quality

## More from the blog

1.  [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 11 Sept 2026 Perceptlayer 7 min read
    
    ## [Web Data Quality Checks for Scraped and API Data](https://perceptlayer.com/blog/web-data-quality-monitoring-checklist)
    
    Seven web data quality checks in Python, with code written against a recorded Zillow search response: schema drift, null rates, duplicate zpids and outliers.
    
2.  [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 20 Oct 2026 Perceptlayer 6 min read
    
    ## [Detect Schema Changes in API JSON Responses](https://perceptlayer.com/blog/web-data-api-json-schema-changes)
    
    Infer a path-and-type schema from a recorded Google Search response (71 paths), diff each new response against it and alert on removed or retyped fields.
    
3.  [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 19 Oct 2026 Perceptlayer 7 min read
    
    ## [Concurrent API Requests in Python: Asyncio Under Rate Limits](https://perceptlayer.com/blog/concurrent-api-requests-python-rate-limit)
    
    A Python asyncio fetcher that paces calls to your per-second limit, honours Retry-After and prices a 20,000-call batch at 20,000 credits.