Blog / AI agents

AI Competitor Monitoring Agent: News, Search and Reviews

7 Jul 2026Perceptlayer7 min read

AI competitor monitoring works best as a scheduled job that collects first and summarises second. This tutorial builds one in Python: once a week, for each company on your list, it pulls Google News articles, Google search results and the latest Trustpilot reviews, keeps only items it hasn't seen before, and has Claude write a digest of what changed with a link for every claim. Each company costs 3 credits a run.

How does the agent work?

It's a fixed pipeline with a model at the end. Code decides what to fetch and what's new; the model only writes. That split matters for monitoring: a model choosing its own searches every week drifts, costs a different amount each run, and can't tell you reliably what's new, while a list of keys on disk can.

1. For each company, fetch news for its name, the search results for its name and its newest Trustpilot reviews.
2. Turn every item into a key: the URL without scheme, www., query string or trailing slash for news and search (split with Python's urllib.parse.urlsplit, 2026), and the review id for reviews.
3. Drop any key already in seen.json, and any repeat within the run.
4. Save the new keys, then send only the new items to Claude with instructions to cite URLs and summarise reviews as themes.
5. Schedule it weekly with cron or any job runner.

You need Python 3.10 or later, a Perceptlayer API key and an Anthropic API key. Get a free API key; verified accounts get 1,000 free credits a month. Install pip install anthropic requests and export PERCEPTLAYER_API_KEY and ANTHROPIC_API_KEY.

Which endpoints does it call?

One per signal, from the Google News API, the Google Search API and the Trustpilot API, each answering a different question about a competitor. News says what's being reported; search says what someone researching the company finds first; reviews say what customers are complaining or happy about this week. All three cost 1 credit per successful call.

The quotes around the company name in q ask Google News for the exact phrase, which cuts unrelated matches for names that are also common words; that's ordinary Google query syntax, which we haven't tested on this endpoint. sort on the reviews endpoint takes relevance or recency; recency is what a weekly diff needs. The Google News API guide and the Trustpilot reviews API guide cover the other parameters.

What do the responses contain?

Each signal comes back with an ID-like field and a date, which is what makes a weekly diff possible. The excerpts below are from recorded responses, trimmed with …. A news result carries the article link, source.name and an exact iso_date alongside the relative date:

{
  "news_results": [
    {
      "position": 1,
      "title": "Top Strategic Technology Trends for 2027",
      "link": "https://www.gartner.com/en/articles/top-technology-trends-2027",
      "type": "article",
      "iso_date": "2026-10-20T15:59:45+00:00",
      "date": "2 days ago",
      "source": { "name": "Gartner", … },
      "story_token": "CBMib0FVX3lxTE1zMGNNN1p0UTdtOXV0U3RoU2JvSGlYUXpQYTZIeEFvcHZGZVpiazBjZ2UyZ0ZhVzZ6NUl3c3Y3LTdCWUVfYVd1YnZuMFdtVElGcHlsR3IwM1BnYXBQNGZueEdldUZJOWM0UnRTQ3Y0WQ",
      …
    },
    …
  ]
}

The Trustpilot response for company_domain=amazon.com carries the company's trustScore and numberOfReviews in businessUnit, then a reviews array. Each review has an id, a rating, title, text, language and dates.publishedDate, plus a consumer object the agent deliberately leaves out:

{
  "businessUnit": {
    "displayName": "Amazon",
    "numberOfReviews": 49830,
    "numberOfReviewsLast12Months": 9156,
    "trustScore": 1.5,
    …
  },
  "reviews": [
    {
      "id": "6adac54ca4e34a0e087ae2b2",
      "rating": 1,
      "language": "de",
      "dates": {
        "experiencedDate": "2026-10-23T00:00:00.000Z",
        "publishedDate": "2026-10-23T04:24:12.000Z"
      },
      …
    },
    …
  ]
}

Use iso_date and publishedDate, not the relative date, when you store items: "2 days ago" means something different next week. The search response's organic_results carry position, title and link, as in the Google Search API guide.

What does the complete agent look like?

Here's the whole script. get adds each response's X-Credits-Used to a running total. collect makes the three calls and flattens them into items. new_items drops what's been seen and updates the state. digest sends the new items to Claude through the Messages API (Anthropic, 2026). Edit COMPANIES to your list; set CLAUDE_MODEL to use another model.

"""Weekly competitor digest: new news, search results and reviews for each company."""
import json
import os
import pathlib
from urllib.parse import urlsplit

import anthropic
import requests

API = "https://api.perceptlayer.com/v1"
HEADERS = {"X-API-Key": os.environ["PERCEPTLAYER_API_KEY"]}
MODEL = os.environ.get("CLAUDE_MODEL", "claude-opus-5-5")
STATE = pathlib.Path("seen.json")
COMPANIES = [
    {"name": "Amazon", "domain": "amazon.com"},
]
spent = 0


def get(path, **params):
    global spent
    r = requests.get(f"{API}/{path}", headers=HEADERS, params=params, timeout=90)
    spent += int(r.headers.get("X-Credits-Used", "0"))
    r.raise_for_status()
    return r.json()


def url_key(url):
    """Same page, same key: no scheme, no www, no query or fragment, no trailing slash."""
    parts = urlsplit(url)
    host = parts.netloc.lower().removeprefix("www.")
    return f"{host}{parts.path.rstrip('/')}"


def collect(company):
    news = get("google-news/news", q=f'"{company["name"]}"', hl="en", gl="us")
    search = get("google-search/search", query=company["name"], hl="en", gl="us")
    reviews = get("trustpilot/company-reviews", company_domain=company["domain"], sort="recency")

    items = []
    for n in news.get("news_results", []):
        items.append({"kind": "news", "key": url_key(n["link"]), "title": n["title"],
                      "source": n.get("source", {}).get("name"), "date": n.get("iso_date"), "url": n["link"]})
    for s in search.get("organic_results", []):
        items.append({"kind": "search", "key": url_key(s["link"]), "title": s["title"],
                      "position": s.get("position"), "url": s["link"]})
    for rv in reviews.get("reviews", []):
        # No reviewer names or profile data go to the model or the digest.
        items.append({"kind": "review", "key": f'trustpilot:{rv["id"]}', "rating": rv["rating"],
                      "title": rv.get("title"), "text": rv.get("text", "")[:500],
                      "language": rv.get("language"), "date": rv.get("dates", {}).get("publishedDate")})
    score = reviews.get("businessUnit", {})
    return items, {"trust_score": score.get("trustScore"), "reviews_total": score.get("numberOfReviews")}


def new_items(company, items, seen):
    known = set(seen.get(company["domain"], []))
    fresh, keys = [], set()
    for it in items:
        if it["key"] in known or it["key"] in keys:
            continue
        keys.add(it["key"])
        fresh.append(it)
    seen[company["domain"]] = sorted(known | keys)
    return fresh


SYSTEM = """You write a weekly competitor digest for a product team.
You get, per company, only items that are new since last week: news articles,
search results and customer reviews. Group by company. For each, say in two or
three bullets what changed, citing the URL of every news or search item you use.
Summarise reviews as themes with counts; never quote a reviewer's name.
If a company has nothing new, say "No change". Do not add facts that are not in the input."""


def digest(report):
    client = anthropic.Anthropic()
    response = client.messages.create(
        model=MODEL,
        max_tokens=16000,
        system=SYSTEM,
        messages=[{"role": "user", "content": json.dumps(report, ensure_ascii=False)}],
    )
    if response.stop_reason == "refusal":
        return "The model declined to write the digest."
    return "".join(b.text for b in response.content if b.type == "text")


def run():
    seen = json.loads(STATE.read_text()) if STATE.exists() else {}
    report = []
    for company in COMPANIES:
        items, score = collect(company)
        fresh = new_items(company, items, seen)
        report.append({"company": company["name"], **score, "new": fresh})
    STATE.write_text(json.dumps(seen))
    return report


if __name__ == "__main__":
    report = run()
    print(digest(report))
    print(f"\nCredits used: {spent}")

Run it with python monitor.py, then add a weekly cron entry such as 0 7 * * 1 cd /path/to/monitor && python monitor.py >> digest.log. The first run reports everything as new; from the second run on, only changes appear. Our guide to scheduling web data pulls compares cron with queues and workflow tools.

What happened when we ran it on recorded data?

We haven't run this script against the live API or a model. We ran run() twice with requests.get mocked to return three recorded responses: Google News for q=technology, Google Search for "best restaurants in Berlin" and Trustpilot reviews for amazon.com. The news and search recordings are for other queries, so this checks the plumbing, not the relevance of results for a real competitor.

The recorded samples are short, so 3 items each; a live response has more. The first run's report, the part sent to the model, was under 3,000 characters for one company. The second run shows the property that matters: the same items on the same keys are dropped, so a quiet week produces "No change" instead of a repeated digest. The Trustpilot score and total came through as 1.5 and 49,830.

URL keys catch the same article twice, but not the same story from two outlets: syndicated wire stories have different URLs. If that noise matters, group news by story_token or by title similarity before the digest. For weighting reviews, see product review data for sentiment analysis.

What does it cost to run?

Three successful calls per company per run, at 1 credit each. Failed calls cost nothing and return X-Credits-Used: 0, so spent counts what was billed, not what was attempted. The credits docs explain billing; the model's tokens are billed separately by Anthropic.

Worked example: 20 companies weekly is 20 × 3 = 60 credits a run, about 260 a month, within the 1,000 free monthly credits. Running the same list daily is 1,800 credits a month, so a Starter pack of 10,000 credits lasts five months; purchased credits last 12 months. See pricing for what credits cost, and budgeting credits per scheduled agent run for caps.

Calls run one after another, so 100 companies is 300 calls, five minutes at the free tier's 1 request a second; the rate limits docs list paid rates, and concurrent API requests in Python shows how to parallelise within them.

How do you make it production-ready?

Five changes take it from script to job. Each is small, and none changes the cost per company.

1. Retry the right errors. Retry rate_limited and temporarily_unavailable after Retry-After; skip a company on not_found and keep going. Handling web data API errors in Python has the pattern, and the errors docs list the codes.
2. Write state only after the digest is sent. Otherwise a failed email loses a week of items.
3. Keep a rank history for search. Store position per URL per week, so the digest can say a page moved from 7 to 2, not only that it's new; search result data for SEO monitoring covers which fields to keep.
4. Track the score. Store trustScore and numberOfReviews each week and report the change.
5. Bound the state file. Drop keys older than a few months, or it grows forever.

For an agent that chooses its own sources, the same endpoints are available through the MCP server, as in the AI agents docs. To start, get a free API key and run a competitor's domain through the playground on the Trustpilot API page. Data is collected from publicly available sources and returned as found at the time of the call.

competitor monitoringai agentsgoogle newstrustpilotpython