--- title: "AI Competitor Monitoring Agent: News, Search and Reviews" description: "A weekly Python agent checks news, Google results and Trustpilot reviews for each competitor, keeps only new URLs and review IDs, and costs 3 credits a company." url: https://perceptlayer.com/blog/ai-competitor-monitoring-agent --- [Blog](https://perceptlayer.com/blog) / [AI agents](https://perceptlayer.com/blog/category/ai-agents) # AI Competitor Monitoring Agent: News, Search and Reviews 7 Jul 2026 Perceptlayer 7 min read AI competitor monitoring works best as a scheduled job that collects first and summarises second. This tutorial builds one in Python: once a week, for each company on your list, it pulls Google News articles, Google search results and the latest Trustpilot reviews, keeps only items it hasn't seen before, and has Claude write a digest of what changed with a link for every claim. Each company costs 3 credits a run. ## Key takeaways - Three calls per company per week: `GET /v1/google-news/news`, `GET /v1/google-search/search` and `GET /v1/trustpilot/company-reviews`, at 1 credit each. - Deduplication happens in code, before the model: news and search items are keyed by a normalised URL, reviews by their Trustpilot `id`, and a JSON file remembers every key. - In a test on recorded responses with mocked HTTP calls, the first run found 9 new items and a second run on the same data found 0, for 3 credits a run. - Reviewer names never reach the model: the agent passes rating, title, text, language and date only. - Monitoring 20 companies weekly costs 20 × 3 × 52 ÷ 12 ≈ 260 credits a month, inside the 1,000 free monthly credits. ## How does the agent work? It's a fixed pipeline with a model at the end. Code decides what to fetch and what's new; the model only writes. That split matters for monitoring: a model choosing its own searches every week drifts, costs a different amount each run, and can't tell you reliably what's new, while a list of keys on disk can. 1. For each company, fetch news for its name, the search results for its name and its newest Trustpilot reviews. 2. Turn every item into a key: the URL without scheme, `www.`, query string or trailing slash for news and search (split with Python's [`urllib.parse.urlsplit`](https://docs.python.org/3/library/urllib.parse.html), 2026), and the review `id` for reviews. 3. Drop any key already in `seen.json`, and any repeat within the run. 4. Save the new keys, then send only the new items to Claude with instructions to cite URLs and summarise reviews as themes. 5. Schedule it weekly with cron or any job runner. You need Python 3.10 or later, a Perceptlayer API key and an Anthropic API key. [Get a free API key](https://perceptlayer.com/signup); verified accounts get 1,000 free credits a month. Install `pip install anthropic requests` and export `PERCEPTLAYER_API_KEY` and `ANTHROPIC_API_KEY`. ## Which endpoints does it call? One per signal, from the [Google News API](https://perceptlayer.com/apis/google-news), the [Google Search API](https://perceptlayer.com/apis/google-search) and the Trustpilot API, each answering a different question about a competitor. News says what's being reported; search says what someone researching the company finds first; reviews say what customers are complaining or happy about this week. All three cost 1 credit per successful call. | Signal | Endpoint | Parameters used | Key for dedupe | | --- | --- | --- | --- | | News | [`/v1/google-news/news`](https://perceptlayer.com/apis/google-news/news) | `q` (the name in quotes), `hl`, `gl` | Article URL | | Search results | [`/v1/google-search/search`](https://perceptlayer.com/apis/google-search/search) | `query`, `hl`, `gl` | Result URL | | Reviews | [`/v1/trustpilot/company-reviews`](https://perceptlayer.com/apis/trustpilot/company-reviews) | `company_domain`, `sort=recency` | Review `id` | The quotes around the company name in `q` ask Google News for the exact phrase, which cuts unrelated matches for names that are also common words; that's ordinary Google query syntax, which we haven't tested on this endpoint. `sort` on the reviews endpoint takes `relevance` or `recency`; recency is what a weekly diff needs. The [Google News API guide](https://perceptlayer.com/blog/google-news-api) and the [Trustpilot reviews API guide](https://perceptlayer.com/blog/trustpilot-reviews-api) cover the other parameters. ## What do the responses contain? Each signal comes back with an ID-like field and a date, which is what makes a weekly diff possible. The excerpts below are from recorded responses, trimmed with `…`. A news result carries the article `link`, `source.name` and an exact `iso_date` alongside the relative `date`: ```json { "news_results": [ { "position": 1, "title": "Top Strategic Technology Trends for 2027", "link": "https://www.gartner.com/en/articles/top-technology-trends-2027", "type": "article", "iso_date": "2026-10-20T15:59:45+00:00", "date": "2 days ago", "source": { "name": "Gartner", … }, "story_token": "CBMib0FVX3lxTE1zMGNNN1p0UTdtOXV0U3RoU2JvSGlYUXpQYTZIeEFvcHZGZVpiazBjZ2UyZ0ZhVzZ6NUl3c3Y3LTdCWUVfYVd1YnZuMFdtVElGcHlsR3IwM1BnYXBQNGZueEdldUZJOWM0UnRTQ3Y0WQ", … }, … ] } ``` The Trustpilot response for `company_domain=amazon.com` carries the company's `trustScore` and `numberOfReviews` in `businessUnit`, then a `reviews` array. Each review has an `id`, a `rating`, `title`, `text`, `language` and `dates.publishedDate`, plus a `consumer` object the agent deliberately leaves out: ```json { "businessUnit": { "displayName": "Amazon", "numberOfReviews": 49830, "numberOfReviewsLast12Months": 9156, "trustScore": 1.5, … }, "reviews": [ { "id": "6adac54ca4e34a0e087ae2b2", "rating": 1, "language": "de", "dates": { "experiencedDate": "2026-10-23T00:00:00.000Z", "publishedDate": "2026-10-23T04:24:12.000Z" }, … }, … ] } ``` Use `iso_date` and `publishedDate`, not the relative `date`, when you store items: "2 days ago" means something different next week. The search response's `organic_results` carry `position`, `title` and `link`, as in the [Google Search API guide](https://perceptlayer.com/blog/google-search-api-json). ## What does the complete agent look like? Here's the whole script. `get` adds each response's `X-Credits-Used` to a running total. `collect` makes the three calls and flattens them into items. `new_items` drops what's been seen and updates the state. `digest` sends the new items to Claude through the [Messages API](https://docs.anthropic.com/en/api/messages) (Anthropic, 2026). Edit `COMPANIES` to your list; set `CLAUDE_MODEL` to use another model. ```python """Weekly competitor digest: new news, search results and reviews for each company.""" import json import os import pathlib from urllib.parse import urlsplit import anthropic import requests API = "https://api.perceptlayer.com/v1" HEADERS = {"X-API-Key": os.environ["PERCEPTLAYER_API_KEY"]} MODEL = os.environ.get("CLAUDE_MODEL", "claude-opus-5-5") STATE = pathlib.Path("seen.json") COMPANIES = [ {"name": "Amazon", "domain": "amazon.com"}, ] spent = 0 def get(path, **params): global spent r = requests.get(f"{API}/{path}", headers=HEADERS, params=params, timeout=90) spent += int(r.headers.get("X-Credits-Used", "0")) r.raise_for_status() return r.json() def url_key(url): """Same page, same key: no scheme, no www, no query or fragment, no trailing slash.""" parts = urlsplit(url) host = parts.netloc.lower().removeprefix("www.") return f"{host}{parts.path.rstrip('/')}" def collect(company): news = get("google-news/news", q=f'"{company["name"]}"', hl="en", gl="us") search = get("google-search/search", query=company["name"], hl="en", gl="us") reviews = get("trustpilot/company-reviews", company_domain=company["domain"], sort="recency") items = [] for n in news.get("news_results", []): items.append({"kind": "news", "key": url_key(n["link"]), "title": n["title"], "source": n.get("source", {}).get("name"), "date": n.get("iso_date"), "url": n["link"]}) for s in search.get("organic_results", []): items.append({"kind": "search", "key": url_key(s["link"]), "title": s["title"], "position": s.get("position"), "url": s["link"]}) for rv in reviews.get("reviews", []): # No reviewer names or profile data go to the model or the digest. items.append({"kind": "review", "key": f'trustpilot:{rv["id"]}', "rating": rv["rating"], "title": rv.get("title"), "text": rv.get("text", "")[:500], "language": rv.get("language"), "date": rv.get("dates", {}).get("publishedDate")}) score = reviews.get("businessUnit", {}) return items, {"trust_score": score.get("trustScore"), "reviews_total": score.get("numberOfReviews")} def new_items(company, items, seen): known = set(seen.get(company["domain"], [])) fresh, keys = [], set() for it in items: if it["key"] in known or it["key"] in keys: continue keys.add(it["key"]) fresh.append(it) seen[company["domain"]] = sorted(known | keys) return fresh SYSTEM = """You write a weekly competitor digest for a product team. You get, per company, only items that are new since last week: news articles, search results and customer reviews. Group by company. For each, say in two or three bullets what changed, citing the URL of every news or search item you use. Summarise reviews as themes with counts; never quote a reviewer's name. If a company has nothing new, say "No change". Do not add facts that are not in the input.""" def digest(report): client = anthropic.Anthropic() response = client.messages.create( model=MODEL, max_tokens=16000, system=SYSTEM, messages=[{"role": "user", "content": json.dumps(report, ensure_ascii=False)}], ) if response.stop_reason == "refusal": return "The model declined to write the digest." return "".join(b.text for b in response.content if b.type == "text") def run(): seen = json.loads(STATE.read_text()) if STATE.exists() else {} report = [] for company in COMPANIES: items, score = collect(company) fresh = new_items(company, items, seen) report.append({"company": company["name"], **score, "new": fresh}) STATE.write_text(json.dumps(seen)) return report if __name__ == "__main__": report = run() print(digest(report)) print(f"\nCredits used: {spent}") ``` Run it with `python monitor.py`, then add a weekly cron entry such as `0 7 * * 1 cd /path/to/monitor && python monitor.py >> digest.log`. The first run reports everything as new; from the second run on, only changes appear. Our guide to [scheduling web data pulls](https://perceptlayer.com/blog/schedule-web-data-pulls-cron-queue) compares cron with queues and workflow tools. ## What happened when we ran it on recorded data? We haven't run this script against the live API or a model. We ran `run()` twice with `requests.get` mocked to return three recorded responses: Google News for `q=technology`, Google Search for "best restaurants in Berlin" and Trustpilot reviews for `amazon.com`. The news and search recordings are for other queries, so this checks the plumbing, not the relevance of results for a real competitor. | Run | New news | New search | New reviews | Credits | | --- | --- | --- | --- | --- | | First | 3 | 3 | 3 | 3 | | Second, same data | 0 | 0 | 0 | 3 | The recorded samples are short, so 3 items each; a live response has more. The first run's report, the part sent to the model, was under 3,000 characters for one company. The second run shows the property that matters: the same items on the same keys are dropped, so a quiet week produces "No change" instead of a repeated digest. The Trustpilot score and total came through as 1.5 and 49,830. URL keys catch the same article twice, but not the same story from two outlets: syndicated wire stories have different URLs. If that noise matters, group news by `story_token` or by title similarity before the digest. For weighting reviews, see [product review data for sentiment analysis](https://perceptlayer.com/blog/product-review-data-sentiment-analysis). ## What does it cost to run? Three successful calls per company per run, at 1 credit each. Failed calls cost nothing and return `X-Credits-Used: 0`, so `spent` counts what was billed, not what was attempted. The [credits docs](https://docs.perceptlayer.com/credits) explain billing; the model's tokens are billed separately by Anthropic. | Companies | Credits a run | Weekly, per month (× 52 ÷ 12) | Daily, per month (× 30) | | --- | --- | --- | --- | | 5 | 15 | 65 | 450 | | 20 | 60 | 260 | 1,800 | | 100 | 300 | 1,300 | 9,000 | Worked example: 20 companies weekly is 20 × 3 = 60 credits a run, about 260 a month, within the 1,000 free monthly credits. Running the same list daily is 1,800 credits a month, so a Starter pack of 10,000 credits lasts five months; purchased credits last 12 months. See [pricing](https://perceptlayer.com/pricing) for what credits cost, and [budgeting credits per scheduled agent run](https://perceptlayer.com/blog/scheduled-agent-credit-budget) for caps. Calls run one after another, so 100 companies is 300 calls, five minutes at the free tier's 1 request a second; the [rate limits docs](https://docs.perceptlayer.com/rate-limits) list paid rates, and [concurrent API requests in Python](https://perceptlayer.com/blog/concurrent-api-requests-python-rate-limit) shows how to parallelise within them. ## How do you make it production-ready? Five changes take it from script to job. Each is small, and none changes the cost per company. 1. **Retry the right errors.** Retry `rate_limited` and `temporarily_unavailable` after `Retry-After`; skip a company on `not_found` and keep going. [Handling web data API errors in Python](https://perceptlayer.com/blog/handle-web-data-api-errors-python) has the pattern, and the [errors docs](https://docs.perceptlayer.com/errors) list the codes. 2. **Write state only after the digest is sent.** Otherwise a failed email loses a week of items. 3. **Keep a rank history for search.** Store `position` per URL per week, so the digest can say a page moved from 7 to 2, not only that it's new; [search result data for SEO monitoring](https://perceptlayer.com/blog/search-result-data-seo-monitoring) covers which fields to keep. 4. **Track the score.** Store `trustScore` and `numberOfReviews` each week and report the change. 5. **Bound the state file.** Drop keys older than a few months, or it grows forever. For an agent that chooses its own sources, the same endpoints are available through the MCP server, as in the [AI agents docs](https://docs.perceptlayer.com/ai-agents). To start, [get a free API key](https://perceptlayer.com/signup) and run a competitor's domain through the playground on the [Trustpilot API page](https://perceptlayer.com/apis/trustpilot). Data is collected from publicly available sources and returned as found at the time of the call. ## APIs in this post - [Google News **1 credit per call** · 1 endpoint](https://perceptlayer.com/apis/google-news) - [Google Search **1 credit per call** · 7 endpoints](https://perceptlayer.com/apis/google-search) - [Trustpilot **1 credit per call** · 6 endpoints](https://perceptlayer.com/apis/trustpilot) ## Frequently asked questions ### Which data sources does the monitoring agent use? Google News articles for each company's name, Google search results for the name and the newest Trustpilot reviews for its domain, through `google-news/news`, `google-search/search` and `trustpilot/company-reviews`. Each returns JSON with a URL or review ID and a date, which the agent uses to keep only items it hasn't seen in earlier runs. ### How much does competitor monitoring cost? 3 credits per company per run, one for each endpoint, at 1 credit per successful call; failed calls cost nothing. Twenty companies weekly is about 260 credits a month, within the 1,000 free monthly credits every verified account gets. Daily runs for the same list are 1,800 credits a month. See [pricing](https://perceptlayer.com/pricing). ### How does the agent avoid reporting the same item twice? It keys news and search results by URL without scheme, `www.`, query string or trailing slash, and reviews by their Trustpilot `id`, and stores every key in `seen.json`. On the next run, any item whose key is already stored is dropped before the model sees it. Syndicated stories with different URLs still get through. ### Does the agent store reviewers' personal data? No. It passes each review's rating, title, text, language and publication date to the model and leaves out the `consumer` object, which holds the reviewer's display name and profile. The system prompt also tells the model never to quote names. Keep it that way if you add fields. ### Why not let the model search for itself? Because a weekly diff needs the same queries every week. A model choosing its own searches varies them, which makes "new since last week" meaningless and the cost per run unpredictable. Code fixes the queries and the cost at 3 credits per company; the model only writes the summary, from items code has already filtered. competitor monitoring ai agents google news trustpilot python ## More from the blog 1. [AI agents](https://perceptlayer.com/blog/category/ai-agents) 23 Oct 2026 Perceptlayer 8 min read ## [Build a Market Research Agent with Search and Trends Data](https://perceptlayer.com/blog/market-research-agent-search-trends) Wire Google Search, Trends and News into one Python agent, cap it at 8 credits per question, and see a recorded Trends response. 2. [AI agents](https://perceptlayer.com/blog/category/ai-agents) 17 Sept 2026 Perceptlayer 8 min read ## [Build a Price Tracking Agent in Python with Live Data](https://perceptlayer.com/blog/price-tracking-agent-python) A Python price tracker that checks Google Shopping and eBay once per product per run, stores history in SQLite, flags drops and costs 2 credits a product. 3. [AI agents](https://perceptlayer.com/blog/category/ai-agents) 11 Sept 2026 Perceptlayer 8 min read ## [OpenAI Agents SDK with a Remote MCP Server for Web Data](https://perceptlayer.com/blog/openai-agents-sdk-mcp-server) Connect the OpenAI Agents SDK to a Streamable HTTP MCP server with a Bearer header, cap call\_endpoint at 3 calls a question, and count credits per run.