Blog / Web data by industry

News Data for Event Detection: Signals and Noise

11 Aug 2026Perceptlayer7 min read

News data detects events when you count independent coverage, not articles. One story syndicated to forty sites is one source repeated; three unrelated outlets writing about the same thing within hours is a signal. So event detection on news search results comes down to four steps: drop duplicate URLs, cluster articles that tell the same story, count distinct sources per cluster in a time window, and compare that count with a baseline.

What counts as an event in news data?

A change in coverage that's larger than normal for that topic. Researchers have formalised this for decades: Jon Kleinberg's burst detection model treats a stream of documents as switching between a baseline state and bursty states, so an event is a period when arrivals come faster than usual (Kleinberg, Bursty and Hierarchical Structure in Streams, 2002). The working version for a monitoring job is simpler: more independent sources than usual, about the same thing, in a short window.

Every part of that sentence filters noise. "Independent" removes syndication. "The same thing" needs clustering. "Short window" needs exact timestamps. "Than usual" needs a baseline per query, because ten articles a day about a large company is quiet, while ten about a small one is news.

What does a news search result give you to work with?

Each result has the fields the four steps need: a link to dedupe on, a title to cluster on, a source name to count, and an exact timestamp to place it in a window. Nothing else is required, which keeps the detector independent of any one source's extras. The Google News endpoint returned this for q=technology, trimmed with …:

{
  "news_results": [
    {
      "position": 1,
      "title": "Top Strategic Technology Trends for 2027",
      "link": "https://www.gartner.com/en/articles/top-technology-trends-2027",
      "type": "article",
      "iso_date": "2026-10-20T15:59:45+00:00",
      "date": "2 days ago",
      "source": { "name": "Gartner", … },
      "story_token": "CBMib0FVX3lxTE1zMGNNN1p0UTdtOXV0U3RoU2JvSGlYUXpQYTZIeEFvcHZGZVpiazBjZ2UyZ0ZhVzZ6NUl3c3Y3LTdCWUVfYVd1YnZuMFdtVElGcHlsR3IwM1BnYXBQNGZueEdldUZJOWM0UnRTQ3Y0WQ",
      "related_publications": [ { "title": "Gartner", "publication_token": "CAAqJQgKIh9DQklTRVFnTWFnMEtDMmRoY25SdVpYSXVZMjl0S0FBUAE", … } ]
    },
    {
      "position": 2,
      "title": "How did AI discover the internet's massive cyber-security risk?",
      "link": "https://www.bbc.com/news/videos/cq0ld7e0jg2eo",
      "iso_date": "2026-10-22T12:17:44+00:00",
      "date": "today",
      "source": { "name": "BBC", … },
      …
    },
    …
  ]
}

The Google News API guide covers the endpoint's parameters, including gl and hl for market and language.

How do you separate the signal from the noise?

Apply the four steps in order, each in plain code. Cheap rules catch most noise: duplicate links and syndicated copies are mechanical problems with mechanical fixes, and only the clustering step involves any judgment. You don't need a language model until then, and often not at all. The functions below were run on the recorded response.

Step 1: Drop duplicate URLs

The same article comes back on consecutive runs and sometimes with tracking parameters. Key each one by host and path, without www. or the query string. In our test, appending ?utm_source=x to the first recorded link produced the same key, and the article was counted once.

Step 2: Cluster articles that tell the same story

Group articles whose titles share most of their words. A Jaccard similarity of title words (shared words divided by all words, after dropping stopwords) is crude but transparent; 0.5 is a reasonable first threshold. On the recording, the three pairs scored 0, 0.083 and 0, so each article is its own cluster: a Gartner trends article, a BBC video on a security risk and a university building ceremony are three stories, not one.

import re
from datetime import datetime, timedelta
from urllib.parse import urlsplit

STOP = {"the", "a", "an", "of", "to", "in", "for", "and", "on", "at", "with", "how", "did", "is", "s"}


def canonical(url):
    p = urlsplit(url)
    return p.netloc.lower().removeprefix("www.") + p.path.rstrip("/")


def words(title):
    return {w for w in re.findall(r"[a-z0-9]+", title.lower()) if w not in STOP}


def similar(a, b):
    """Jaccard similarity of title words."""
    wa, wb = words(a), words(b)
    return len(wa & wb) / len(wa | wb) if wa | wb else 0.0


def clusters(articles, threshold=0.5):
    """Greedy clustering: an article joins the first cluster whose lead title is similar enough."""
    seen, groups = set(), []
    for a in sorted(articles, key=lambda x: x["iso_date"]):
        key = canonical(a["link"])
        if key in seen:
            continue                                   # same article twice
        seen.add(key)
        for g in groups:
            if similar(g[0]["title"], a["title"]) >= threshold:
                g.append(a)
                break
        else:
            groups.append([a])
    return groups

Title matching misses rewrites with different headlines and can merge unrelated stories with formulaic titles. When that matters, compare embeddings of title plus snippet instead; the structure of the job stays the same.

Step 3: Count distinct sources in a window

For each cluster, count the distinct source.name values among articles whose iso_date falls inside the window. Distinct sources, not articles: a wire story republished by many outlets often keeps its headline, so it lands in one cluster, but if it was carried under one source name it still counts once.

def signals(articles, now, window=timedelta(hours=24), min_sources=3):
    """A cluster is an event when enough distinct sources covered it inside the window."""
    out = []
    for g in clusters(articles):
        recent = [a for a in g if now - datetime.fromisoformat(a["iso_date"]) <= window]
        sources = {a["source"]["name"] for a in recent}
        out.append({"lead": g[0]["title"], "articles": len(g), "sources_in_window": len(sources),
                    "event": len(sources) >= min_sources})
    return out

Step 4: Compare with a baseline

A fixed min_sources works for narrow queries. For broad ones, keep each query's daily count of distinct sources and flag a day only when it's well above that query's recent norm, for example at least three times the median of the previous 14 days, with a floor of 3. That's a rough version of the baseline-versus-burst idea; it needs a couple of weeks of history before it says anything.

What does the worked example show?

We ran signals on the recorded response with the clock set to 18:00 UTC on 22 October 2026, the day of the BBC article, a 24-hour window and a threshold of 3 distinct sources. It returned three clusters of one article each, none of them an event, which is what a broad query over unrelated stories should produce:

That's the right answer for a broad query like "technology": unrelated stories, one source each, two of them outside the 24-hour window. The recorded sample has three results, so it can show the mechanics but not a burst; a real event would show up as one cluster with several sources inside the window.

The example also shows why the query matters more than the threshold. "technology" matches everything, so no single story stands out. Event detection works on narrow queries: a company, a product, a place, a regulation. Those have a low, stable baseline that a real event clearly exceeds.

When is a larger news dataset a better fit?

When you need global coverage rather than a watch list. The GDELT Project monitors print, broadcast and web news in over 100 languages and updates its event data every 15 minutes, free (GDELT, 2026). For research across all world news, or for event types rather than named entities, that's a better starting point than running thousands of searches.

A search endpoint fits the opposite case: a defined list of companies, products or places you care about, checked on a schedule, with results you can act on and link to. For a weekly digest built on the same results, see our AI competitor monitoring agent.

What does an event detector cost to run?

One call per query per check, at 1 credit per successful call; failed calls cost nothing and return X-Credits-Used: 0. The cost is set by how many queries you watch and how often. The credits docs explain billing.

Worked example: 50 queries every 6 hours is 50 × 4 × 30 = 6,000 credits a month, inside a Starter pack of 10,000 credits; hourly, it's 36,000, inside a Build pack of 50,000 credits. Ten queries checked twice a day, 600 credits, fit in the 1,000 free monthly credits. See pricing for what credits cost; for running the checks on a timer, see scheduling web data pulls.

To try a query, get a free API key and run it in the playground on the Google News API page. Data is collected from publicly available sources and returned as found at the time of the call.

event detectionnews datagoogle newsmonitoringpython