Blog / Web data by industry

Search Result Data for SEO Monitoring: Fields to Track

10 Sept 2026Perceptlayer8 min read

For SEO monitoring with SERP data, store one row per keyword, per day, per market and device: the organic position, the rank_absolute, the title, the link and the snippet. Add the related questions and related searches as a second table. At 1 credit per call, 200 keywords checked daily cost 6,000 credits a month, and the fields below tell you why rankings moved.

Which SERP data fields matter for SEO monitoring?

Five fields carry most of the signal: position, rank_absolute, title, link and snippet. Position tells you where a page ranks among organic results, the link tells you who holds it, and the title and snippet show how that page presents itself. Everything else either explains movement or is noise for rank tracking.

Here is what the Search endpoint returns for each organic result, taken from the recorded response for best restaurants in Berlin.

Two things about this table. redirect_link also comes back, but it is a Google redirect URL, so use link as the key. And snippet_highlighted_words shows which words matched the query, which is handy for content audits but optional for monitoring.

Why store both position and absolute rank?

In the recorded response, position numbers the organic results 1, 2 and so on, while rank_absolute is a page-wide number that the local results share. A result can hold position 1 and still sit well down the page, and only the second number shows it.

The recorded response makes this concrete. The local results carry rank_absolute 1 and 2, and the first organic result carries 4. This excerpt is trimmed with …:

{
  "position": 1,
  "title": "The Best Restaurants in Berlin - Food Guide & Map",
  "link": "https://andershusa.com/best-restaurants-berlin-germany-guide/",
  "snippet": "Dec 3, 2025 — Tim Raue is the most internationally renowned restaurant in Berlin. …",
  "rank_group": 1,
  "rank_absolute": 4
}

Position 1, absolute 4. If you only stored position, you could see a flat line while clicks fell because modules sit above the result. Storing both gives you a way to tell a ranking loss from a layout change.

The same response lists local_results and serp_results entries with their own rank_absolute values. Counting how many modules sit above your result each day gives you a layout trend with no extra calls.

What should you store per keyword per day?

Store one row per keyword per day per market, with the organic results as child rows. The key is the date, the keyword and the parameters that change the page: gl (country), hl (language) and device. Leave any of them out and you'll compare a desktop page in one country with a mobile page in another.

A workable layout has three tables.

Keep every organic result, not just your own. Competitor rows are what turn a drop from "we lost position 3" into "a new domain took position 3 with a rewritten title". Store the domain as its own column, parsed from link, so a group-by is cheap.

Don't store redirect_link or the pagination links; they're long, change per call and add nothing to analysis. Keep the raw JSON in cold storage only if you want to re-parse later. For loading JSON into a warehouse, see loading API JSON into Postgres.

How do related questions fit into rank monitoring?

Related questions show which follow-up queries Google attaches to a keyword, and they change over time. The Search response already includes related_questions and related_searches, so you can log them in the same call that returns your rankings. Each question is a content idea, and a new one appearing is a signal worth reading.

From the recorded response:

"related_questions": [
  {
    "question": "What is the coolest restaurant in Berlin?",
    "next_page_token": "fc=EqEBCmJBSmlUNHRMWjV0…"
  },
  …
]

Store the question text and diff it against yesterday's set. The next_page_token is only useful if you want to expand a question with the Related Questions endpoint, which takes q, next_page_token, hl and gl and costs 1 credit per call. Expanding every question for every keyword multiplies your cost, so do it for a short list of keywords that matter most.

For a view of how interest in a topic moves over longer periods, pair this with search trend data. For the endpoint itself, the Google Search API guide covers the parameters.

How do you pull the data for 200 keywords?

Loop over your keyword list, call GET /v1/google-search/search once per keyword with query, gl, hl and device, and write the fields above into SQLite or your warehouse. The script below does that for a list in keywords.txt, one keyword per line, retries a 429 or 503 up to three times, and logs and skips any keyword that still fails.

import os
import sqlite3
import time
from datetime import date, datetime, timezone
from urllib.parse import urlparse

import requests

BASE = "https://api.perceptlayer.com"
HEADERS = {"X-API-Key": os.environ["API_KEY"]}
MARKET = {"gl": "us", "hl": "en", "device": "desktop"}
MAX_ATTEMPTS = 3

db = sqlite3.connect("serp.db")
db.executescript("""
CREATE TABLE IF NOT EXISTS checks (
  id INTEGER PRIMARY KEY,
  day TEXT, keyword TEXT, gl TEXT, hl TEXT, device TEXT,
  organic_results_count INTEGER, fetched_at TEXT,
  UNIQUE (day, keyword, gl, hl, device));
CREATE TABLE IF NOT EXISTS results (
  check_id INTEGER REFERENCES checks(id),
  position INTEGER, rank_absolute INTEGER,
  title TEXT, link TEXT, domain TEXT, snippet TEXT,
  PRIMARY KEY (check_id, position));
CREATE TABLE IF NOT EXISTS questions (
  check_id INTEGER REFERENCES checks(id),
  kind TEXT, text TEXT,
  PRIMARY KEY (check_id, kind, text));
""")


def search(keyword):
    """Return the parsed response, or None after logging a failure."""
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            r = requests.get(
                f"{BASE}/v1/google-search/search",
                params={"query": keyword, **MARKET},
                headers=HEADERS,
                timeout=60,
            )
        except requests.RequestException as e:
            print(f"{keyword!r}: attempt {attempt} failed: {type(e).__name__}")
            time.sleep(5)
            continue
        if r.status_code in (429, 503):
            print(f"{keyword!r}: HTTP {r.status_code}, attempt {attempt}")
            time.sleep(int(r.headers.get("Retry-After", "5")))
            continue
        if not r.ok:
            # 403 restricted_endpoint, 402 insufficient_credits and so on:
            # retrying won't help, so log it and move to the next keyword.
            print(f"{keyword!r}: HTTP {r.status_code}, skipping")
            return None
        return r.json()
    print(f"{keyword!r}: gave up after {MAX_ATTEMPTS} attempts")
    return None


today = date.today().isoformat()
with open("keywords.txt") as f:
    keywords = [line.strip() for line in f if line.strip()]

failed = []
for kw in keywords:
    data = search(kw)
    if data is None:
        failed.append(kw)
        continue
    key = (today, kw, MARKET["gl"], MARKET["hl"], MARKET["device"])
    organic = data.get("organic_results", [])
    with db:  # one transaction per keyword
        # A rerun replaces today's rows for this key instead of doubling them.
        old = db.execute(
            "SELECT id FROM checks WHERE day=? AND keyword=? AND gl=? "
            "AND hl=? AND device=?", key).fetchone()
        if old:
            db.execute("DELETE FROM results WHERE check_id=?", old)
            db.execute("DELETE FROM questions WHERE check_id=?", old)
            db.execute("DELETE FROM checks WHERE id=?", old)
        check_id = db.execute(
            "INSERT INTO checks (day, keyword, gl, hl, device, "
            "organic_results_count, fetched_at) VALUES (?,?,?,?,?,?,?)",
            (*key, len(organic), datetime.now(timezone.utc).isoformat()),
        ).lastrowid
        for item in organic:
            link = item.get("link", "")
            db.execute(
                "INSERT OR REPLACE INTO results VALUES (?,?,?,?,?,?,?)",
                (check_id, item.get("position"), item.get("rank_absolute"),
                 item.get("title"), link, urlparse(link).netloc,
                 item.get("snippet")),
            )
        for q in data.get("related_questions", []):
            db.execute("INSERT OR IGNORE INTO questions VALUES (?,?,?)",
                       (check_id, "related_questions", q.get("question")))
        for s in data.get("related_searches", []):
            db.execute("INSERT OR IGNORE INTO questions VALUES (?,?,?)",
                       (check_id, "related_searches", s.get("query")))

print(f"done: {len(keywords) - len(failed)} stored, {len(failed)} failed")
for kw in failed:
    print("retry later:", kw)

Rate limits are per account, counted per minute. A free account gets 1 request a second, so 200 sequential keywords take at least 200 seconds there; paying accounts run at least 5 a second. The rate limits page has the details, and concurrent requests under a rate limit shows how to parallelise. Run it daily with cron or a queue; scheduling web data pulls compares the options.

A request whose reply was lost might have succeeded and been billed, because requests have no idempotency key. For a daily snapshot that's a small risk, but the script deletes and rewrites a key's rows, so a rerun on the same day replaces them rather than doubling them. A 403 with restricted_endpoint is not retried: if a call returns it, accept the restricted-endpoint terms in the dashboard and rerun. The script logs the failed keywords at the end, so you can rerun the list once the cause is fixed.

What does tracking 200 keywords daily cost?

At 1 credit per successful call, 200 keywords checked once a day cost 200 credits a day: 200 × 30 = 6,000 credits for a 30-day month, or 200 × 365 = 73,000 credits for a year. Failed calls cost nothing and return X-Credits-Used: 0, so retries on errors don't add to the total.

The weekly row is 200 × 4 = 800 credits for four checks in a month. The free tier gives 1,000 credits every month after email verification, which covers that weekly setup, or 5 days of the daily one (1,000 ÷ 200).

The Starter pack is 10,000 credits. The first row (6,000 a month) fits inside it, but the second row, at 12,000, does not. Purchased credits last 12 months, and there's no subscription, so a pack is not a monthly commitment. Pricing shows what credits cost, and the credits page explains how they're debited. Every response carries X-Credits-Remaining, so log it and alert before the balance runs out; an empty balance returns insufficient_credits (402) and nothing more is debited.

Be honest about the trade-offs. Daily checks on every keyword are rarely needed. Daily for your top 50 and weekly for the long tail gets most of the value at a fraction of the credits. And rank data from a single call is a snapshot of the page as returned at that moment. Compare trends over weeks, not a single day's move, and run the data quality checks before you alert on a drop.

For what Google says about how titles and snippets are chosen, read its guidance on snippets in search results, which explains why your stored snippets will change without you touching the page. For first-party click and impression data, Search Console's performance report is the primary source and complements this: it tells you your own traffic, while the monitoring above tells you who else is on the page.

To try it, get a free API key, then run one keyword in the playground on the Google Search API page before you loop over all 200.

seoserprank trackinggoogle searchmonitoring