--- title: "Search Result Data for SEO Monitoring: Fields to Track" description: "Which search result fields to store per keyword per day, from position to snippet, and what 200 keywords tracked daily cost in credits." url: https://perceptlayer.com/blog/search-result-data-seo-monitoring --- [Blog](https://perceptlayer.com/blog) / [Web data by industry](https://perceptlayer.com/blog/category/web-data) # Search Result Data for SEO Monitoring: Fields to Track 10 Sept 2026 Perceptlayer 8 min read For SEO monitoring with SERP data, store one row per keyword, per day, per market and device: the organic `position`, the `rank_absolute`, the `title`, the `link` and the `snippet`. Add the related questions and related searches as a second table. At 1 credit per call, 200 keywords checked daily cost 6,000 credits a month, and the fields below tell you why rankings moved. ## Key takeaways - Store `position` and `rank_absolute` separately. In the recorded response, the first organic result has `position` 1 but `rank_absolute` 4, with local results at `rank_absolute` 1 and 2 above it. - Keep `title` and `snippet` every day. A rank change with no title change points at the page's competition; a title change points at a rewrite, while a snippet can change without any page edit. - Key every row by date, keyword, `gl`, `hl` and `device`, or day-to-day comparisons mix different result pages. - 200 keywords × 1 credit × 30 days = 6,000 credits a month. The 1,000 free monthly credits cover 5 days of that. - The same search response carries `related_questions` and `related_searches`, so tracking them adds no calls. ## Which SERP data fields matter for SEO monitoring? Five fields carry most of the signal: `position`, `rank_absolute`, `title`, `link` and `snippet`. Position tells you where a page ranks among organic results, the link tells you who holds it, and the title and snippet show how that page presents itself. Everything else either explains movement or is noise for rank tracking. Here is what the [Search endpoint](https://perceptlayer.com/apis/google-search/search) returns for each organic result, taken from the recorded response for `best restaurants in Berlin`. | Field | What it holds | Why you store it | | --- | --- | --- | | `position` | Rank among organic results | The number you report | | `rank_group` | A second rank number; it equals `position` in the recorded organic results | Store it if you want to compare with other result types | | `rank_absolute` | A page-wide rank that also numbers other modules | Shows how far down the page a result sits | | `title` | The result's title | Detects rewrites and competitor changes | | `link` | The ranking URL | Tells you which of your pages ranks, and which rival | | `displayed_link` | The breadcrumb-style URL shown | Useful for spotting site structure changes | | `snippet` | The text shown under the title | Detects snippet changes and date stamps | | `source` | The site name | Quick grouping by publisher | Two things about this table. `redirect_link` also comes back, but it is a Google redirect URL, so use `link` as the key. And `snippet_highlighted_words` shows which words matched the query, which is handy for content audits but optional for monitoring. ## Why store both position and absolute rank? In the recorded response, `position` numbers the organic results 1, 2 and so on, while `rank_absolute` is a page-wide number that the local results share. A result can hold position 1 and still sit well down the page, and only the second number shows it. The recorded response makes this concrete. The local results carry `rank_absolute` 1 and 2, and the first organic result carries 4. This excerpt is trimmed with `…`: ```json { "position": 1, "title": "The Best Restaurants in Berlin - Food Guide & Map", "link": "https://andershusa.com/best-restaurants-berlin-germany-guide/", "snippet": "Dec 3, 2025 — Tim Raue is the most internationally renowned restaurant in Berlin. …", "rank_group": 1, "rank_absolute": 4 } ``` Position 1, absolute 4. If you only stored `position`, you could see a flat line while clicks fell because modules sit above the result. Storing both gives you a way to tell a ranking loss from a layout change. The same response lists `local_results` and `serp_results` entries with their own `rank_absolute` values. Counting how many modules sit above your result each day gives you a layout trend with no extra calls. ## What should you store per keyword per day? Store one row per keyword per day per market, with the organic results as child rows. The key is the date, the keyword and the parameters that change the page: `gl` (country), `hl` (language) and `device`. Leave any of them out and you'll compare a desktop page in one country with a mobile page in another. A workable layout has three tables. | Table | Key | Columns | | --- | --- | --- | | `checks` | date, keyword, `gl`, `hl`, `device` (unique) | `organic_results_count`, fetched time | | `results` | check, `position` | `rank_absolute`, `title`, `link`, domain, `snippet` | | `questions` | check, kind, text | kind ( `related_questions` or `related_searches`) and the question or search text | Keep every organic result, not just your own. Competitor rows are what turn a drop from "we lost position 3" into "a new domain took position 3 with a rewritten title". Store the domain as its own column, parsed from `link`, so a group-by is cheap. Don't store `redirect_link` or the pagination links; they're long, change per call and add nothing to analysis. Keep the raw JSON in cold storage only if you want to re-parse later. For loading JSON into a warehouse, see [loading API JSON into Postgres](https://perceptlayer.com/blog/load-web-data-json-into-postgres). ## How do related questions fit into rank monitoring? Related questions show which follow-up queries Google attaches to a keyword, and they change over time. The Search response already includes `related_questions` and `related_searches`, so you can log them in the same call that returns your rankings. Each question is a content idea, and a new one appearing is a signal worth reading. From the recorded response: ```json "related_questions": [ { "question": "What is the coolest restaurant in Berlin?", "next_page_token": "fc=EqEBCmJBSmlUNHRMWjV0…" }, … ] ``` Store the `question` text and diff it against yesterday's set. The `next_page_token` is only useful if you want to expand a question with the [Related Questions endpoint](https://perceptlayer.com/apis/google-related-questions/related-questions), which takes `q`, `next_page_token`, `hl` and `gl` and costs 1 credit per call. Expanding every question for every keyword multiplies your cost, so do it for a short list of keywords that matter most. For a view of how interest in a topic moves over longer periods, pair this with [search trend data](https://perceptlayer.com/blog/search-trend-data-demand-forecasting). For the endpoint itself, the [Google Search API guide](https://perceptlayer.com/blog/google-search-api-json) covers the parameters. ## How do you pull the data for 200 keywords? Loop over your keyword list, call `GET /v1/google-search/search` once per keyword with `query`, `gl`, `hl` and `device`, and write the fields above into SQLite or your warehouse. The script below does that for a list in `keywords.txt`, one keyword per line, retries a 429 or 503 up to three times, and logs and skips any keyword that still fails. ```python import os import sqlite3 import time from datetime import date, datetime, timezone from urllib.parse import urlparse import requests BASE = "https://api.perceptlayer.com" HEADERS = {"X-API-Key": os.environ["API_KEY"]} MARKET = {"gl": "us", "hl": "en", "device": "desktop"} MAX_ATTEMPTS = 3 db = sqlite3.connect("serp.db") db.executescript(""" CREATE TABLE IF NOT EXISTS checks ( id INTEGER PRIMARY KEY, day TEXT, keyword TEXT, gl TEXT, hl TEXT, device TEXT, organic_results_count INTEGER, fetched_at TEXT, UNIQUE (day, keyword, gl, hl, device)); CREATE TABLE IF NOT EXISTS results ( check_id INTEGER REFERENCES checks(id), position INTEGER, rank_absolute INTEGER, title TEXT, link TEXT, domain TEXT, snippet TEXT, PRIMARY KEY (check_id, position)); CREATE TABLE IF NOT EXISTS questions ( check_id INTEGER REFERENCES checks(id), kind TEXT, text TEXT, PRIMARY KEY (check_id, kind, text)); """) def search(keyword): """Return the parsed response, or None after logging a failure.""" for attempt in range(1, MAX_ATTEMPTS + 1): try: r = requests.get( f"{BASE}/v1/google-search/search", params={"query": keyword, **MARKET}, headers=HEADERS, timeout=60, ) except requests.RequestException as e: print(f"{keyword!r}: attempt {attempt} failed: {type(e).__name__}") time.sleep(5) continue if r.status_code in (429, 503): print(f"{keyword!r}: HTTP {r.status_code}, attempt {attempt}") time.sleep(int(r.headers.get("Retry-After", "5"))) continue if not r.ok: # 403 restricted_endpoint, 402 insufficient_credits and so on: # retrying won't help, so log it and move to the next keyword. print(f"{keyword!r}: HTTP {r.status_code}, skipping") return None return r.json() print(f"{keyword!r}: gave up after {MAX_ATTEMPTS} attempts") return None today = date.today().isoformat() with open("keywords.txt") as f: keywords = [line.strip() for line in f if line.strip()] failed = [] for kw in keywords: data = search(kw) if data is None: failed.append(kw) continue key = (today, kw, MARKET["gl"], MARKET["hl"], MARKET["device"]) organic = data.get("organic_results", []) with db: # one transaction per keyword # A rerun replaces today's rows for this key instead of doubling them. old = db.execute( "SELECT id FROM checks WHERE day=? AND keyword=? AND gl=? " "AND hl=? AND device=?", key).fetchone() if old: db.execute("DELETE FROM results WHERE check_id=?", old) db.execute("DELETE FROM questions WHERE check_id=?", old) db.execute("DELETE FROM checks WHERE id=?", old) check_id = db.execute( "INSERT INTO checks (day, keyword, gl, hl, device, " "organic_results_count, fetched_at) VALUES (?,?,?,?,?,?,?)", (*key, len(organic), datetime.now(timezone.utc).isoformat()), ).lastrowid for item in organic: link = item.get("link", "") db.execute( "INSERT OR REPLACE INTO results VALUES (?,?,?,?,?,?,?)", (check_id, item.get("position"), item.get("rank_absolute"), item.get("title"), link, urlparse(link).netloc, item.get("snippet")), ) for q in data.get("related_questions", []): db.execute("INSERT OR IGNORE INTO questions VALUES (?,?,?)", (check_id, "related_questions", q.get("question"))) for s in data.get("related_searches", []): db.execute("INSERT OR IGNORE INTO questions VALUES (?,?,?)", (check_id, "related_searches", s.get("query"))) print(f"done: {len(keywords) - len(failed)} stored, {len(failed)} failed") for kw in failed: print("retry later:", kw) ``` Rate limits are per account, counted per minute. A free account gets 1 request a second, so 200 sequential keywords take at least 200 seconds there; paying accounts run at least 5 a second. The [rate limits page](https://docs.perceptlayer.com/rate-limits) has the details, and [concurrent requests under a rate limit](https://perceptlayer.com/blog/concurrent-api-requests-python-rate-limit) shows how to parallelise. Run it daily with cron or a queue; [scheduling web data pulls](https://perceptlayer.com/blog/schedule-web-data-pulls-cron-queue) compares the options. A request whose reply was lost might have succeeded and been billed, because requests have no idempotency key. For a daily snapshot that's a small risk, but the script deletes and rewrites a key's rows, so a rerun on the same day replaces them rather than doubling them. A `403` with `restricted_endpoint` is not retried: if a call returns it, accept the restricted-endpoint terms in the dashboard and rerun. The script logs the failed keywords at the end, so you can rerun the list once the cause is fixed. ## What does tracking 200 keywords daily cost? At 1 credit per successful call, 200 keywords checked once a day cost 200 credits a day: 200 × 30 = 6,000 credits for a 30-day month, or 200 × 365 = 73,000 credits for a year. Failed calls cost nothing and return `X-Credits-Used: 0`, so retries on errors don't add to the total. | Setup | Calls a day | Credits a month (30 days) | | --- | --- | --- | | 200 keywords, one market and device | 200 | 6,000 | | 200 keywords, desktop and mobile | 400 | 12,000 | | 200 keywords, plus Related Questions on 20 of them | 220 | 6,600 | | 200 keywords, checked weekly | about 29 | about 800 | The weekly row is 200 × 4 = 800 credits for four checks in a month. The free tier gives 1,000 credits every month after email verification, which covers that weekly setup, or 5 days of the daily one (1,000 ÷ 200). The Starter pack is 10,000 credits. The first row (6,000 a month) fits inside it, but the second row, at 12,000, does not. Purchased credits last 12 months, and there's no subscription, so a pack is not a monthly commitment. [Pricing](https://perceptlayer.com/pricing) shows what credits cost, and the [credits page](https://docs.perceptlayer.com/credits) explains how they're debited. Every response carries `X-Credits-Remaining`, so log it and alert before the balance runs out; an empty balance returns `insufficient_credits` (402) and nothing more is debited. Be honest about the trade-offs. Daily checks on every keyword are rarely needed. Daily for your top 50 and weekly for the long tail gets most of the value at a fraction of the credits. And rank data from a single call is a snapshot of the page as returned at that moment. Compare trends over weeks, not a single day's move, and run the [data quality checks](https://perceptlayer.com/blog/web-data-quality-monitoring-checklist) before you alert on a drop. For what Google says about how titles and snippets are chosen, read its guidance on [snippets in search results](https://developers.google.com/search/docs/appearance/snippet), which explains why your stored snippets will change without you touching the page. For first-party click and impression data, [Search Console's performance report](https://support.google.com/webmasters/answer/7042828) is the primary source and complements this: it tells you your own traffic, while the monitoring above tells you who else is on the page. To try it, [get a free API key](https://perceptlayer.com/signup), then run one keyword in the playground on the [Google Search API page](https://perceptlayer.com/apis/google-search) before you loop over all 200. ## APIs in this post - [Google Search **1 credit per call** · 7 endpoints](https://perceptlayer.com/apis/google-search) - [Google Related Questions **1 credit per call** · 1 endpoint](https://perceptlayer.com/apis/google-related-questions) ## Frequently asked questions ### What is the minimum I should store for rank tracking? Store the date, keyword, `gl`, `hl`, `device`, and for each organic result the `position`, `rank_absolute`, `link`, `title` and `snippet`. That is enough to plot rank, spot a competitor taking over, and see rewrites. Add related questions if you also want to track which follow-up questions appear for a keyword. ### How many credits does tracking 200 keywords every day use? 200 keywords × 1 credit × 30 days = 6,000 credits a month, or 73,000 a year at 200 × 365. Desktop and mobile together double it to 12,000 a month. Failed calls are not billed, and the free tier of 1,000 monthly credits covers 5 days of daily checks. ### Do I need a separate call for related questions? Not to read them. The Search response includes `related_questions` and `related_searches`, as the recorded response shows. A separate call to the Related Questions endpoint, at 1 credit, is only needed to expand a question using its `next_page_token`, or to look up questions for a query directly with `q`. ### Why does my position differ from what I see in the browser? Results vary with country, language, device and the time of the call. Pass `gl`, `hl` and `device` explicitly so every day's check uses the same settings, and treat the data as returned at the time of the call. Compare your series with a browser check in the same market, not a personalised session. ### What happens if a daily run hits the rate limit? A 429 response carries a `Retry-After` header with the seconds to wait; the script above sleeps for that long and retries, up to three attempts, then logs the keyword and moves on. Limits are per account and counted per minute. Retried 429s are errors, so they aren't billed. See the [errors page](https://docs.perceptlayer.com/errors) for every code. seo serp rank tracking google search monitoring ## More from the blog 1. [Web data by industry](https://perceptlayer.com/blog/category/web-data) 11 Aug 2026 Perceptlayer 7 min read ## [News Data for Event Detection: Signals and Noise](https://perceptlayer.com/blog/news-data-event-detection) Turn news search results into event signals: drop duplicate URLs, cluster syndicated titles, count distinct sources in a window, and compare with a baseline. 2. [API guides](https://perceptlayer.com/blog/category/api-guides) 16 Jul 2026 Perceptlayer 7 min read ## [Google AI Overview API: Get AI Overviews as JSON](https://perceptlayer.com/blog/google-ai-overview-api) Google has no AI Overview API. GET /v1/google-search/search-ai-overview takes a query, hl and gl, costs 1 credit, and suits daily keyword tracking. 3. [Web data by industry](https://perceptlayer.com/blog/category/web-data) 21 Oct 2026 Perceptlayer 8 min read ## [Job Posting Data for Labour-Market Research](https://perceptlayer.com/blog/job-posting-data-labour-market-research) Which Indeed posting fields support trend analysis, where postings mislead versus hires, and a weekly collection plan that costs 3,000 credits a month.