Blog / Data engineering

Scheduling Web Data Pulls: Cron, Queues or Workflows

7 Oct 2026Perceptlayer10 min read

Use cron if one script finishes the whole pull inside its schedule window, add a job queue once you fan out to hundreds of calls or need per-item retries, and move to a workflow engine only when steps depend on each other and must survive restarts. Failure handling and cost control decide it, not scheduling syntax. This article compares the three on a Google Trends pull and gives a verdict per team size.

What are you actually scheduling?

A recurring web data pull is a loop of API calls on a clock: for each keyword, place or product, call an endpoint, store the result, and do it again tomorrow. The scheduler only decides when the loop starts. What separates cron, queues and workflows is what happens when call number 87 of 200 fails.

Take a concrete job. You track search interest for 200 keywords with GET /v1/google-trends/interest, which takes q, plus optional geo, time_range, hl and search_type. Each successful call costs 1 credit; failed calls cost nothing and return X-Credits-Used: 0. A daily run is 200 credits.

This is the same shape as most scheduled pulls: prices, listings, reviews, rankings. The Google Trends API is a good test case because the calls are cheap, independent and repeated for the same keys every day.

How do cron, queues and workflows compare?

Cron runs a script on a fixed schedule and forgets it, a job queue stores each unit of work and retries it independently, and a workflow engine stores the whole run's progress as durable state. Cron is cheapest to operate, a workflow engine is safest for long multi-step runs, and a queue sits between them on both counts.

None of the three removes the need to design for repeats. Queues deliver work more than once under some failure modes, cron can start a second run before the first ends, and a workflow retries a step that may already have succeeded. You handle that in your data model, not in the scheduler.

Which failures should every scheduled pull handle?

Every scheduled pull has to handle five outcomes: a rate limit, a temporary outage, a client timeout, a bad request and an empty balance. Each needs a different response, and only the first two should retry automatically. The error reference lists the codes; here is how each one should behave in a scheduled job.

The timeout row matters most. Requests carry no idempotency key, so a request whose reply was lost may have succeeded and been billed. The pattern in retrying API calls without paying twice applies to all three schedulers: write a row keyed by keyword, geography and time window, and check it before you call.

The 402 row is why a run needs a stop condition. Retrying 200 jobs against an empty balance wastes time and fills your logs. For the full set of status handling in code, see handling web data API errors in Python.

When is cron enough?

Cron is enough when the whole pull finishes well inside its schedule interval and a failed run can simply be rerun. For 200 calls at the free tier's 1 request a second, a run takes a little over three minutes of requests. With a paying account's minimum of 5 requests a second (300 a minute), the same run takes about 40 seconds.

The script below pulls interest for a keyword list and stores each result in SQLite. It writes an unknown row before each call and marks it ok only when the response arrives. A rerun skips finished keywords and skips unknown ones too, listing them for you to review, because a timed-out call may have been billed. It honours Retry-After and stops on 402.

import os, sqlite3, sys, time, datetime
import requests

URL = "https://api.perceptlayer.com/v1/google-trends/interest"
KEYWORDS = ["tesla", "rivian", "lucid"]
GEO = "US"
PAUSE = 1.0  # seconds between calls; 1 request a second on the free tier
TIMEOUT = object()  # sentinel: the reply was lost, the call may have been billed

db = sqlite3.connect("trends.db")
db.execute("""CREATE TABLE IF NOT EXISTS pulls (
    keyword TEXT, geo TEXT, day TEXT, status TEXT, fetched_at TEXT, body TEXT,
    PRIMARY KEY (keyword, geo, day))""")

day = datetime.date.today().isoformat()
headers = {"X-API-Key": os.environ["API_KEY"]}

def fetch(keyword):
    params = {"q": keyword, "geo": GEO, "time_range": "1y"}
    for attempt in range(4):
        try:
            r = requests.get(URL, params=params, headers=headers, timeout=30)
        except requests.Timeout:
            return TIMEOUT
        # every response carries the balance; log it to resolve "unknown" rows later
        print(f"{r.status_code} {keyword} credits_remaining={r.headers.get('X-Credits-Remaining')}")
        if r.status_code == 200:
            return r.text
        if r.status_code == 402:
            release(keyword)
            sys.exit("insufficient_credits: stopping the run")
        if r.status_code in (429, 503):
            time.sleep(min(int(r.headers.get("Retry-After", "5")), 60))
            continue
        err = r.json().get("error", {})
        print(f"{r.status_code} {err.get('code')} {keyword} {err.get('request_id')}")
        if r.status_code == 502:
            time.sleep(5)
            continue
        return None
    return None

def release(keyword):
    # an error response was not billed, so the attempt can be retried later
    db.execute("DELETE FROM pulls WHERE keyword=? AND geo=? AND day=? AND status='unknown'",
               (keyword, GEO, day))
    db.commit()

for kw in KEYWORDS:
    row = db.execute("SELECT status FROM pulls WHERE keyword=? AND geo=? AND day=?",
                     (kw, GEO, day)).fetchone()
    if row and row[0] == "ok":
        continue
    if row:
        print(f"review {kw}: earlier attempt has no recorded outcome, skipping")
        continue
    # record the attempt before calling, so a timeout or crash leaves a trace
    db.execute("INSERT INTO pulls VALUES (?,?,?,?,?,NULL)",
               (kw, GEO, day, "unknown", datetime.datetime.utcnow().isoformat()))
    db.commit()
    body = fetch(kw)
    if body is TIMEOUT:
        print(f"timeout {kw}: left as unknown, may have been billed")
    elif body:
        db.execute("UPDATE pulls SET status='ok', body=? WHERE keyword=? AND geo=? AND day=?",
                   (body, kw, GEO, day))
        db.commit()
    else:
        release(kw)
    time.sleep(PAUSE)

Schedule it with a lock so a slow run can't overlap the next one:

0 6 * * * flock -n /tmp/trends.lock python3 /opt/pulls/pull.py >> /var/log/trends.log 2>&1

Set API_KEY in the environment of the cron job. The response for each keyword is a JSON document with timeline_data and interest_over_time; to load such documents into a database, see loading API JSON into Postgres.

Cron's weak spot is visibility. A run that dies at 3 a.m. leaves a log line and nothing else, so add an alert on a missing ok row for today, not on the exit code. Rows stuck at unknown need a person: compare the X-Credits-Remaining values you logged before and after the call, then delete the row to allow a retry if no credit was taken.

When does a job queue earn its place?

A job queue earns its place when a pull is large enough that one slow or failing item shouldn't hold up the rest, or when volume is spiky. Each keyword becomes a message, a consumer makes one call per message, and a failure retries that message alone. The cost is a queue, a consumer process and a dead-letter destination to watch.

Queues also give you one control cron doesn't: consumer concurrency. Rate limits are per account and shared by all its keys, counted per minute, so ten consumers running flat out compete for the same budget. Cap the consumer count to your account's rate, and read RateLimit-Remaining and RateLimit-Reset if you want to slow down before you hit a 429. The rate limits docs have the exact headers, and concurrent API requests in Python shows a bounded-concurrency client.

The decision logic of a consumer is small. This function maps a response to what the queue should do with the message:

def decide(status, headers):
    """Return (action, delay_seconds) for a queue message."""
    if status == 200:
        return "ack", 0
    if status in (429, 503):
        return "retry", int(headers.get("Retry-After", "5"))
    if status == 502:
        return "retry_limited", 30
    if status == 402:
        return "halt", 0          # pause the consumers, not just this job
    return "dead_letter", 0       # 401, 403, 404, 422: retrying won't help

Most queues deliver at least once, so a message can arrive twice. Amazon documents this behaviour for standard queues (Amazon SQS, standard queues). The consumer must therefore do the same check the cron script does: look up keyword, geography and window in your results table before calling the API. A duplicate delivery then costs a database read, not a credit.

When do you need a workflow engine?

You need a workflow engine when a run has several dependent steps, takes long enough that a restart in the middle is likely, or has to be auditable step by step. The engine persists each step's result, so a crashed worker resumes after the last completed step instead of repeating paid calls.

The Google Trends endpoints show where dependency appears. GET /v1/google-trends/trending-news requires a news_id and a geo, so a pull that follows up on trending items has a second stage whose input comes from the first. A pipeline might then fetch, validate, load and notify, with different retry rules at each stage.

Engines such as Temporal record a workflow's event history and replay it to recover state after a failure (Temporal documentation, Workflows), and they apply a retry policy to each activity (Temporal documentation, Retry Policies). That is real value for multi-stage runs. For a single loop of independent calls it is overhead: you run and monitor an engine to get what a results table and a queue already give you.

The best test is to ask whether a failure at step three should resume at step three. If the answer is "just rerun everything, it's cheap", you don't need a workflow engine yet. A 200-credit run is cheaper to repeat than a cluster is to operate.

What does each option cost in credits?

Scheduling doesn't change the price of a call, but it changes how many calls you make and how many you repeat. Cost is calls × 1 credit, so the schedule frequency and the retry discipline are the two levers. Credit packs and what they cost are on the pricing page; here the numbers are in credits only.

The daily pull is above the 1,000 free credits every month, so it needs a pack: 6,000 credits fits the Starter pack's 10,000. The 6-hourly pull of 24,000 fits the Build pack's 50,000, and the hourly pull of 144,000 fits the Growth pack's 200,000. Purchased credits last 12 months, with no subscription.

Before raising frequency, check whether the data changes that fast. Weekly-bucketed interest values, as in the recorded timeline_data points ("date": "Oct 19 – 25, 2025"), change slowly: past weeks are fixed buckets, and only the current week's value can move between hourly runs. Caching and TTLs are covered in caching API responses, and per-run budgets in budgeting credits per scheduled run.

Retries cost nothing when they follow an error response, because failed calls are free. The expensive repeat is the timeout: the call may have succeeded and been billed, so a blind retry pays twice. Your results table is the cost control.

Which one should your team pick?

Pick by how many people run the pipeline and how many steps it has, not by how fashionable the tool is. A solo developer or a team of two should start with cron. A small team with fan-out or deadlines should add a queue. A platform team running many pipelines with dependent stages is where a workflow engine pays back.

Migration is gentle in this order. The results table and the decide logic carry over unchanged from cron to a queue, and the queue's per-item steps become workflow activities later. If you start with the most complex option, you pay its operating cost from day one.

If you are weighing a scheduled API against a scraper you maintain, what maintaining a scraper involves covers that comparison. To try the pull above, get a free API key (1,000 free credits a month, no credit card) and run the cron script against three keywords first.

schedulingcronjob queueworkflowsgoogle trendsretries