--- title: "Scheduling Web Data Pulls: Cron, Queues or Workflows" description: "Cron, a job queue or a workflow engine for recurring API pulls? Compare failure handling and credit cost, with a verdict per team size and a cron script." url: https://perceptlayer.com/blog/schedule-web-data-pulls-cron-queue --- [Blog](https://perceptlayer.com/blog) / [Data engineering](https://perceptlayer.com/blog/category/data-engineering) # Scheduling Web Data Pulls: Cron, Queues or Workflows 7 Oct 2026 Perceptlayer 10 min read Use cron if one script finishes the whole pull inside its schedule window, add a job queue once you fan out to hundreds of calls or need per-item retries, and move to a workflow engine only when steps depend on each other and must survive restarts. Failure handling and cost control decide it, not scheduling syntax. This article compares the three on a Google Trends pull and gives a verdict per team size. ## Key takeaways - Cron costs nothing to run and is enough for a pull of a few hundred calls per run, as long as the script skips work it has already done. - A queue turns each call into its own retryable job, so one `429` or `503` delays one keyword, not the whole run. - A workflow engine records progress between steps, which pays off when a later call needs the result of an earlier one. - Every pattern needs your own record of what was fetched: requests have no idempotency key, so a timed-out call may have been billed. - A daily pull of 200 keywords at 1 credit each is 6,000 credits a month; the same pull hourly is 144,000. ## What are you actually scheduling? A recurring web data pull is a loop of API calls on a clock: for each keyword, place or product, call an endpoint, store the result, and do it again tomorrow. The scheduler only decides when the loop starts. What separates cron, queues and workflows is what happens when call number 87 of 200 fails. Take a concrete job. You track search interest for 200 keywords with `GET /v1/google-trends/interest`, which takes `q`, plus optional `geo`, `time_range`, `hl` and `search_type`. Each successful call costs 1 credit; failed calls cost nothing and return `X-Credits-Used: 0`. A daily run is 200 credits. This is the same shape as most scheduled pulls: prices, listings, reviews, rankings. The [Google Trends API](https://perceptlayer.com/apis/google-trends) is a good test case because the calls are cheap, independent and repeated for the same keys every day. ## How do cron, queues and workflows compare? Cron runs a script on a fixed schedule and forgets it, a job queue stores each unit of work and retries it independently, and a workflow engine stores the whole run's progress as durable state. Cron is cheapest to operate, a workflow engine is safest for long multi-step runs, and a queue sits between them on both counts. | | Cron | Job queue | Workflow engine | | --- | --- | --- | --- | | Unit of retry | The whole run, or whatever your script retries inline | One job (one keyword) | One step, with its own retry policy | | State between failures | Only what you write to a file or database | Queue contents plus your results table | Recorded by the engine | | Rate-limit handling | Sleep inside the script | Delay the one job, cap consumer concurrency | Retry policy with a backoff per step | | Overlapping runs | You prevent them (a lock) | Natural: jobs interleave | Each run is a separate instance | | Extra infrastructure | None | A queue and a consumer | An engine, workers and a history store | | Cost visibility | Count calls in the script | Count completed jobs | Count completed steps | | Fits | One script, one table | Fan-out, spiky volume | Dependent steps, long or auditable runs | None of the three removes the need to design for repeats. Queues deliver work more than once under some failure modes, cron can start a second run before the first ends, and a workflow retries a step that may already have succeeded. You handle that in your data model, not in the scheduler. ## Which failures should every scheduled pull handle? Every scheduled pull has to handle five outcomes: a rate limit, a temporary outage, a client timeout, a bad request and an empty balance. Each needs a different response, and only the first two should retry automatically. The [error reference](https://docs.perceptlayer.com/errors) lists the codes; here is how each one should behave in a scheduled job. | Response | Meaning | What the job does | | --- | --- | --- | | `429` `rate_limited` | Rate limited; carries `Retry-After` | Wait `Retry-After` seconds, retry | | `503` `temporarily_unavailable` | Temporarily unavailable; carries `Retry-After` | Wait `Retry-After` seconds, retry | | `502` `source_error` | An error from the source; no `Retry-After` is documented | Retry a few times with a delay, then give up on this key | | Client timeout | You don't know if it was billed | Treat as unknown; check your own record before repeating | | `422` `invalid_parameters` | The request is wrong | Don't retry; fix the input | | `402` `insufficient_credits` | Prepaid balance is empty | Stop the whole run; nothing more is debited | The timeout row matters most. Requests carry no idempotency key, so a request whose reply was lost may have succeeded and been billed. The pattern in [retrying API calls without paying twice](https://perceptlayer.com/blog/retry-web-api-calls-without-paying-twice) applies to all three schedulers: write a row keyed by keyword, geography and time window, and check it before you call. The `402` row is why a run needs a stop condition. Retrying 200 jobs against an empty balance wastes time and fills your logs. For the full set of status handling in code, see [handling web data API errors in Python](https://perceptlayer.com/blog/handle-web-data-api-errors-python). ## When is cron enough? Cron is enough when the whole pull finishes well inside its schedule interval and a failed run can simply be rerun. For 200 calls at the free tier's 1 request a second, a run takes a little over three minutes of requests. With a paying account's minimum of 5 requests a second (300 a minute), the same run takes about 40 seconds. The script below pulls interest for a keyword list and stores each result in SQLite. It writes an `unknown` row before each call and marks it `ok` only when the response arrives. A rerun skips finished keywords and skips `unknown` ones too, listing them for you to review, because a timed-out call may have been billed. It honours `Retry-After` and stops on `402`. ```python import os, sqlite3, sys, time, datetime import requests URL = "https://api.perceptlayer.com/v1/google-trends/interest" KEYWORDS = ["tesla", "rivian", "lucid"] GEO = "US" PAUSE = 1.0 # seconds between calls; 1 request a second on the free tier TIMEOUT = object() # sentinel: the reply was lost, the call may have been billed db = sqlite3.connect("trends.db") db.execute("""CREATE TABLE IF NOT EXISTS pulls ( keyword TEXT, geo TEXT, day TEXT, status TEXT, fetched_at TEXT, body TEXT, PRIMARY KEY (keyword, geo, day))""") day = datetime.date.today().isoformat() headers = {"X-API-Key": os.environ["API_KEY"]} def fetch(keyword): params = {"q": keyword, "geo": GEO, "time_range": "1y"} for attempt in range(4): try: r = requests.get(URL, params=params, headers=headers, timeout=30) except requests.Timeout: return TIMEOUT # every response carries the balance; log it to resolve "unknown" rows later print(f"{r.status_code} {keyword} credits_remaining={r.headers.get('X-Credits-Remaining')}") if r.status_code == 200: return r.text if r.status_code == 402: release(keyword) sys.exit("insufficient_credits: stopping the run") if r.status_code in (429, 503): time.sleep(min(int(r.headers.get("Retry-After", "5")), 60)) continue err = r.json().get("error", {}) print(f"{r.status_code} {err.get('code')} {keyword} {err.get('request_id')}") if r.status_code == 502: time.sleep(5) continue return None return None def release(keyword): # an error response was not billed, so the attempt can be retried later db.execute("DELETE FROM pulls WHERE keyword=? AND geo=? AND day=? AND status='unknown'", (keyword, GEO, day)) db.commit() for kw in KEYWORDS: row = db.execute("SELECT status FROM pulls WHERE keyword=? AND geo=? AND day=?", (kw, GEO, day)).fetchone() if row and row[0] == "ok": continue if row: print(f"review {kw}: earlier attempt has no recorded outcome, skipping") continue # record the attempt before calling, so a timeout or crash leaves a trace db.execute("INSERT INTO pulls VALUES (?,?,?,?,?,NULL)", (kw, GEO, day, "unknown", datetime.datetime.utcnow().isoformat())) db.commit() body = fetch(kw) if body is TIMEOUT: print(f"timeout {kw}: left as unknown, may have been billed") elif body: db.execute("UPDATE pulls SET status='ok', body=? WHERE keyword=? AND geo=? AND day=?", (body, kw, GEO, day)) db.commit() else: release(kw) time.sleep(PAUSE) ``` Schedule it with a lock so a slow run can't overlap the next one: ```bash 0 6 * * * flock -n /tmp/trends.lock python3 /opt/pulls/pull.py >> /var/log/trends.log 2>&1 ``` Set `API_KEY` in the environment of the cron job. The response for each keyword is a JSON document with `timeline_data` and `interest_over_time`; to load such documents into a database, see [loading API JSON into Postgres](https://perceptlayer.com/blog/load-web-data-json-into-postgres). Cron's weak spot is visibility. A run that dies at 3 a.m. leaves a log line and nothing else, so add an alert on a missing `ok` row for today, not on the exit code. Rows stuck at `unknown` need a person: compare the `X-Credits-Remaining` values you logged before and after the call, then delete the row to allow a retry if no credit was taken. ## When does a job queue earn its place? A job queue earns its place when a pull is large enough that one slow or failing item shouldn't hold up the rest, or when volume is spiky. Each keyword becomes a message, a consumer makes one call per message, and a failure retries that message alone. The cost is a queue, a consumer process and a dead-letter destination to watch. Queues also give you one control cron doesn't: consumer concurrency. Rate limits are per account and shared by all its keys, counted per minute, so ten consumers running flat out compete for the same budget. Cap the consumer count to your account's rate, and read `RateLimit-Remaining` and `RateLimit-Reset` if you want to slow down before you hit a `429`. The [rate limits docs](https://docs.perceptlayer.com/rate-limits) have the exact headers, and [concurrent API requests in Python](https://perceptlayer.com/blog/concurrent-api-requests-python-rate-limit) shows a bounded-concurrency client. The decision logic of a consumer is small. This function maps a response to what the queue should do with the message: ```python def decide(status, headers): """Return (action, delay_seconds) for a queue message.""" if status == 200: return "ack", 0 if status in (429, 503): return "retry", int(headers.get("Retry-After", "5")) if status == 502: return "retry_limited", 30 if status == 402: return "halt", 0 # pause the consumers, not just this job return "dead_letter", 0 # 401, 403, 404, 422: retrying won't help ``` Most queues deliver at least once, so a message can arrive twice. Amazon documents this behaviour for standard queues ( [Amazon SQS, standard queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html)). The consumer must therefore do the same check the cron script does: look up keyword, geography and window in your results table before calling the API. A duplicate delivery then costs a database read, not a credit. ## When do you need a workflow engine? You need a workflow engine when a run has several dependent steps, takes long enough that a restart in the middle is likely, or has to be auditable step by step. The engine persists each step's result, so a crashed worker resumes after the last completed step instead of repeating paid calls. The Google Trends endpoints show where dependency appears. `GET /v1/google-trends/trending-news` requires a `news_id` and a `geo`, so a pull that follows up on trending items has a second stage whose input comes from the first. A pipeline might then fetch, validate, load and notify, with different retry rules at each stage. Engines such as Temporal record a workflow's event history and replay it to recover state after a failure ( [Temporal documentation, Workflows](https://docs.temporal.io/workflows)), and they apply a retry policy to each activity ( [Temporal documentation, Retry Policies](https://docs.temporal.io/encyclopedia/retry-policies)). That is real value for multi-stage runs. For a single loop of independent calls it is overhead: you run and monitor an engine to get what a results table and a queue already give you. The best test is to ask whether a failure at step three should resume at step three. If the answer is "just rerun everything, it's cheap", you don't need a workflow engine yet. A 200-credit run is cheaper to repeat than a cluster is to operate. ## What does each option cost in credits? Scheduling doesn't change the price of a call, but it changes how many calls you make and how many you repeat. Cost is calls × 1 credit, so the schedule frequency and the retry discipline are the two levers. Credit packs and what they cost are on the [pricing page](https://perceptlayer.com/pricing); here the numbers are in credits only. | Pull | Calls per run | Runs per month | Credits per month | | --- | --- | --- | --- | | 200 keywords, daily | 200 | 30 | 200 × 30 = 6,000 | | 200 keywords, every 6 hours | 200 | 120 | 200 × 120 = 24,000 | | 200 keywords, hourly | 200 | 720 | 200 × 720 = 144,000 | The daily pull is above the 1,000 free credits every month, so it needs a pack: 6,000 credits fits the Starter pack's 10,000. The 6-hourly pull of 24,000 fits the Build pack's 50,000, and the hourly pull of 144,000 fits the Growth pack's 200,000. Purchased credits last 12 months, with no subscription. Before raising frequency, check whether the data changes that fast. Weekly-bucketed interest values, as in the recorded `timeline_data` points ( `"date": "Oct 19 – 25, 2025"`), change slowly: past weeks are fixed buckets, and only the current week's value can move between hourly runs. Caching and TTLs are covered in [caching API responses](https://perceptlayer.com/blog/cache-web-data-api-responses), and per-run budgets in [budgeting credits per scheduled run](https://perceptlayer.com/blog/scheduled-agent-credit-budget). Retries cost nothing when they follow an error response, because failed calls are free. The expensive repeat is the timeout: the call may have succeeded and been billed, so a blind retry pays twice. Your results table is the cost control. ## Which one should your team pick? Pick by how many people run the pipeline and how many steps it has, not by how fashionable the tool is. A solo developer or a team of two should start with cron. A small team with fan-out or deadlines should add a queue. A platform team running many pipelines with dependent stages is where a workflow engine pays back. | Situation | Verdict | | --- | --- | | Solo or two people, under a few hundred calls per run | Cron with a results table and a lock | | Small team, hundreds to thousands of calls per run, need per-item retries | Queue with capped consumers | | Spiky load (many keywords added at once) | Queue; cron would overrun its window | | Dependent stages, long runs, audit trail required | Workflow engine | | Many teams sharing pipelines and on-call | Workflow engine, with queues inside steps if fan-out is large | | An agent triggers pulls on demand | Queue or direct calls; see the [MCP docs](https://docs.perceptlayer.com/ai-agents) | Migration is gentle in this order. The results table and the `decide` logic carry over unchanged from cron to a queue, and the queue's per-item steps become workflow activities later. If you start with the most complex option, you pay its operating cost from day one. If you are weighing a scheduled API against a scraper you maintain, [what maintaining a scraper involves](https://perceptlayer.com/blog/scraper-vs-api-maintenance-cost) covers that comparison. To try the pull above, [get a free API key](https://perceptlayer.com/signup) (1,000 free credits a month, no credit card) and run the cron script against three keywords first. ## APIs in this post - [Google Trends **1 credit per call** · 6 endpoints](https://perceptlayer.com/apis/google-trends) ## Frequently asked questions ### How often should I schedule an API data pull? As often as the data changes and no more. The recorded Google Trends interest response buckets values by week, so hourly pulls of a one-year window repeat the same numbers. Daily is a common start. At 1 credit per call, 200 keywords daily is 6,000 credits a month; hourly is 144,000. ### What happens to a scheduled run when my credits run out? Calls return `insufficient_credits` (402) and nothing more is debited. There is no automatic top-up, so your job must treat 402 as a stop signal, not a retry. Alert on it, and check `X-Credits-Remaining`, which every response carries, before large runs. ### Can I run pulls in parallel to finish faster? Yes, within your account's request rate. Limits are per account, shared by all keys, and counted per minute: 1 request a second on the free tier, and at least 5 a second (300 a minute) on a paying account. Parallel workers beyond that only produce `429` responses, so cap concurrency to your rate. ### Is a retry after a timeout safe? Not automatically. Requests have no idempotency key, so a call whose reply was lost may have succeeded and been billed. Record each fetch by keyword, geography and window in your own table, and check it before repeating. Retries after `429` and `503` are safe because those calls are not billed. scheduling cron job queue workflows google trends retries ## More from the blog 1. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 22 Sept 2026 Perceptlayer 7 min read ## [Handle Web Data API Errors in Python: 4xx, 5xx and Timeouts](https://perceptlayer.com/blog/handle-web-data-api-errors-python) Map the API's error codes to retry, fix or stop in Python: 429 and 503 honour Retry-After, 4xx stops, and failed calls cost 0 credits. 2. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 15 Aug 2026 Perceptlayer 11 min read ## [Scraper or API: What Maintaining a Scraper Really Involves](https://perceptlayer.com/blog/scraper-vs-api-maintenance-cost) What a scraper costs to keep running: selector fixes, proxies, retries and monitoring, set against an API at 1 credit per successful call. 3. [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 31 Jul 2026 Perceptlayer 10 min read ## [Retry Web API Calls Without Paying Twice](https://perceptlayer.com/blog/retry-web-api-calls-without-paying-twice) Retry only 429, 502 and 503. Failed responses return X-Credits-Used: 0, but a timeout is the gap. The credit arithmetic, a Python loop and a balance check.