---
title: "Scraper or API: What Maintaining a Scraper Really Involves"
description: "What a scraper costs to keep running: selector fixes, proxies, retries and monitoring, set against an API at 1 credit per successful call."
url: https://perceptlayer.com/blog/scraper-vs-api-maintenance-cost
---

[Blog](https://perceptlayer.com/blog) / [Data engineering](https://perceptlayer.com/blog/category/data-engineering)

# Scraper or API: What Maintaining a Scraper Really Involves

15 Aug 2026 Perceptlayer 11 min read

A scraper is cheap to write and expensive to keep alive. The first version takes an afternoon; the upkeep is a recurring tax of selector repairs, blocked requests, retry logic and monitoring that never reaches zero. An API replaces that tax with a per-call price. For most teams who need data from many websites, the API is cheaper once you count engineer hours. A scraper still wins in a few specific cases, listed below.

## Key takeaways

-   A scraper's real cost is recurring work: fixing selectors after layout changes, handling blocks and proxies, writing retries, and monitoring for silent breakage.
-   Price the upkeep in engineer hours. At an assumed $100 an hour, 8 hours a month is $800 of upkeep. Count your API usage separately in credits, at 1 credit per call, and check [pricing](https://perceptlayer.com/pricing) for what they cost.
-   Perceptlayer covers 248 websites and 1,659 endpoints with one key, and failed calls cost nothing and return `X-Credits-Used: 0`.
-   Build a scraper when the source isn't in any catalogue, when you need fields an API doesn't return, or when volume is so high that per-call pricing exceeds upkeep.
-   Moving off a scraper is low risk: run both for a week, diff the fields, then switch.

## What does maintaining a scraper involve week to week?

A scraper breaks in four recurring ways: the page layout changes and your selectors stop matching, the site starts rejecting your requests, transient errors need retries, and nobody notices a partial failure until a downstream report looks wrong. Each is small on its own. Together they become a standing job that belongs to someone, whether or not it's on their title.

### Selector breakage

Your scraper reads a page by pointing at structure: a CSS selector, an XPath, a position in the DOM. The site owner never agreed to keep that structure stable, and a redesign, an A/B test or a renamed class takes your parser down. The failure is often quiet. The request returns 200, the selector matches nothing, and you store an empty field.

That quiet part is what costs time. A loud crash gets fixed the same day. An empty `price` column gets found when someone asks why the dashboard dropped.

### Blocks, proxies and rate limits

Sites limit how fast one client can read them. Once you scale past a few requests a minute you meet 429 responses, CAPTCHAs and IP-based limits. Teams respond with proxy pools, request pacing and header tuning, and each of those has its own bill and its own failure modes.

Respect the site's rules while you do this. The [Robots Exclusion Protocol (RFC 9309)](https://www.rfc-editor.org/rfc/rfc9309) is the baseline for what a crawler should read, and a site's terms can add limits on top. Engineering around a site's protections is a legal and ethical question as much as a technical one, and it's the part of scraper upkeep that gets least attention in estimates.

### Retries and error handling

Every network call fails sometimes. A scraper needs a policy: which statuses to retry, how long to wait, when to stop, and how to avoid hammering a site that is already struggling. The HTTP spec defines [`Retry-After`](https://www.rfc-editor.org/rfc/rfc9110#field.retry-after) for exactly this, but a scraper only sees it if the site sends it. Our guide to [retrying web API calls without paying twice](https://perceptlayer.com/blog/retry-web-api-calls-without-paying-twice) covers the pattern in detail.

### Monitoring

You can't maintain what you can't see. A scraper in production needs a success rate per site, a check that each field is populated, an alert when output volume drops, and a way to replay a failed job. Building that is real work, and it has to be maintained too.

## Where does the engineering time actually go?

The time goes to recurring tasks, not to the original build. Count the tasks per site, because they multiply with every source you add: ten sites means ten sets of selectors, ten sets of block behaviour and ten places for a silent failure. The table lists what each task costs you in practice.

| Recurring task | What triggers it | How it shows up | Scales with |
| --- | --- | --- | --- |
| Fix selectors | Layout change, A/B test, renamed class | Empty or wrong fields, often with HTTP 200 | Number of sites |
| Handle blocks | Request rate, repeated patterns | 403, 429, CAPTCHAs, timeouts | Request volume |
| Run proxies | Blocks, geo targeting | A separate vendor bill and a new failure source | Request volume |
| Write retries | Transient errors, rate limits | Duplicate work or gaps in the data | Number of sites |
| Monitor output | Silent breakage | A report that looks wrong days later | Number of sites |
| Handle pagination | Cursor or page changes | Missing or repeated pages | Number of endpoints |

None of this is exotic. It's ordinary engineering, and the issue is that it never finishes. If you run pagination loops against anything, see [paginating a web data API in Python](https://perceptlayer.com/blog/paginate-web-data-api-python) for the safe version.

## How does the cost compare with per-call pricing?

Price a scraper by multiplying hours by your loaded hourly rate, then compare it with calls times the per-call price. The only figure you need from us is the credits per call. Perceptlayer gives 1,000 free credits every month after you verify an email address, and sells prepaid packs with no subscription and no seats. The [pricing](https://perceptlayer.com/pricing) page shows what credits cost.

| Pack | Credits | Included rate (requests a second) |
| --- | --- | --- |
| Starter | 10,000 | 5 |
| Build | 50,000 | 5 |
| Growth | 200,000 | 5 |
| Pro | 500,000 | 5 |
| Scale | 1,000,000 | 10 |
| Volume | 5,000,000 | 20 |
| Volume 10M | 10,000,000 | 25 |
| Volume 25M | 25,000,000 | 25 |
| Volume 50M | 50,000,000 | 50 |

A successful call costs the credits shown on its endpoint, which is 1 credit for nearly all of them. Failed calls cost nothing. Purchased credits last 12 months.

### A worked example

Take 100,000 calls a month at 1 credit each: 100,000 calls × 1 credit = 100,000 credits a month. That fits inside one Growth pack of 200,000 credits, which lasts about two months at that rate.

Now the scraper side. The hours below are an assumption, not a measurement; replace them with your own. Say one engineer spends 8 hours a month on selector fixes, block handling and monitoring, at an assumed $100 an hour. That's $800 a month, before proxy fees and servers.

To decide which is cheaper, use the break-even form:

monthly calls × 1 credit = credits a month; price those credits on the pricing page and compare with monthly upkeep

Work in credits first. At 1 credit per call, your monthly calls are your monthly credits. Then check the [pricing](https://perceptlayer.com/pricing) page for what that many credits cost in the pack you'd buy. Set that against your upkeep, which is $800 in this example. If the credits come to less than the upkeep, the API is cheaper. If they come to more, the scraper is cheaper, as long as upkeep stays where it is.

The comparison flips when volume is very high and upkeep is low, because per-call cost grows with volume and upkeep mostly doesn't. Do the sum with your own numbers.

## Scraper vs API: how do they compare side by side?

An in-house scraper gives you control over every field and no per-call fee, at the price of owning every failure. An API gives you a stable JSON contract and a predictable bill, at the price of being limited to the endpoints and fields it offers. The table compares the two across the things that decide maintenance.

| Concern | In-house scraper | Perceptlayer API |
| --- | --- | --- |
| Output shape | Whatever your parser extracts | JSON with named fields from every endpoint |
| Layout changes | Your selectors break, you fix them | Not your parser |
| Blocks and proxies | Yours to manage and pay for | Not part of your code |
| Retries | You write and tune the policy | `Retry-After` on 429 and 503; you still write the loop |
| Failure signal | Often silent | HTTP status plus `{"error": {"code", "message", "request_id"}}` |
| Cost model | Engineer hours, proxies, servers | Credits per successful call; failed calls cost nothing |
| Coverage | Any page you can read | 248 websites, 1,659 endpoints |
| Custom fields | Anything on the page | Only what the endpoint returns |
| Spend visibility | Infrastructure bills | `X-Credits-Used` and `X-Credits-Remaining` on every successful response |

An API doesn't remove error handling. You still write a retry loop and decide what to do with a 502 `source_error`. What it removes is the part that depends on someone else's HTML. See the [errors reference](https://docs.perceptlayer.com/errors) and the [rate limits](https://docs.perceptlayer.com/rate-limits) for what to handle.

Data is collected from publicly available sources and returned as found at the time of the call. That means you should validate it the way you would any third-party feed.

## Which should your team choose?

Choose by team size and by how many sources you need, because maintenance cost scales with sources while an API's cost scales with calls. The verdicts below are plain recommendations, not rules. They assume the data you need is in the catalogue; check the [list of APIs](https://perceptlayer.com/apis) first.

| Situation | Verdict |
| --- | --- |
| Solo developer or small team, a few sources, no one to own upkeep | Use an API. You'd be your own on-call for the scraper. |
| Team building an AI agent that needs many sources | Use an API, over HTTP or [MCP](https://docs.perceptlayer.com/ai-agents). One key beats a scraper per site. |
| Data team with a stable list of sources and a dedicated engineer | Start with an API and keep a scraper only for gaps. |
| Platform team with one source at very high volume | Cost both. A scraper may win if upkeep is low and volume is huge. |
| Source isn't in any catalogue | Build a scraper. |

### Small teams

If nobody owns the scraper, nobody fixes it. The silent failures arrive at the worst time, and the person who wrote the parser has moved on. A per-call price is easier to defend than an engineer's time spent on selectors.

### Agent builders

Agents call many sources, and a scraper per source multiplies upkeep. One key and a catalogue you can search cuts that to a single integration. The MCP server at `https://api.perceptlayer.com/mcp` has `search_endpoints` and `get_endpoint` (both free) and `call_endpoint`, which spends credits on success only. Cap the spend with the approach in [AI agent API cost control](https://perceptlayer.com/blog/ai-agent-web-data-costs).

### Larger data teams

You have the people to run a scraper. The question is whether you should. Put the first few sources on the API, measure the hours you get back, and keep scrapers where the catalogue has no coverage. The [data engineering hub](https://perceptlayer.com/blog/category/data-engineering) has related pipeline guidance.

## When is a scraper the right call?

Build a scraper when the data isn't available any other way, when you need fields an API doesn't expose, or when your volume makes per-call pricing the larger cost. Those cases are real, and an article that pretended otherwise wouldn't help you. Each comes with a condition.

-   **The source isn't covered.** If no catalogue lists the website, you have no choice. Check [the APIs list](https://perceptlayer.com/apis) before you start.
-   **You need page-specific fields.** An API returns the fields its endpoint defines. If you need something visible on the page but absent from the JSON, a scraper can read it.
-   **A website offers its own official API that fits.** Use it. An official API is a contract from the site owner, and that's the most stable source you can get.
-   **Volume dwarfs upkeep.** Once your monthly calls pass the break-even point from the worked example, against one stable page, a scraper's fixed upkeep can undercut per-call pricing.
-   **You need exact control over timing and raw HTML.** Archiving pages or replaying parsers over stored HTML needs the HTML itself.

Even then, keep it small. Scrape only what you're allowed to, keep the parser narrow, and log which fields came back empty.

## How do you move from a scraper to an API?

Run both side by side for a week, compare the fields, then switch one source at a time. The comparison tells you whether the API covers the fields you actually use. You need a free key and one call per source to find out.

1.  [Get a free API key](https://perceptlayer.com/signup). Keys start with `sk_live_`, and the free tier gives 1,000 credits a month with no credit card.
2.  Send the key in the `X-API-Key` header, as in [the authentication docs](https://docs.perceptlayer.com/authentication).
3.  Pick the endpoint that matches your scraper's output and run it in the playground on that API's page.
4.  Diff the fields against your scraper's output for the same inputs. Note which fields are missing.
5.  Log `X-Credits-Used` and `X-Credits-Remaining` from each successful response so the cost is visible from day one.
6.  Switch the source, then retire its selectors and proxy rules.

Here is a copyable call. Replace `ENDPOINT_PATH` with the path shown on the endpoint's page, and add that endpoint's parameters as the page lists them ( [request format](https://docs.perceptlayer.com/requests)):

```bash
curl -i "https://api.perceptlayer.com/ENDPOINT_PATH" \
  -H "X-API-Key: $API_KEY"
```

The `-i` flag prints the response headers, so you can read `X-Credits-Used` and `X-Credits-Remaining`. The response body shape depends on the endpoint, so check the fields on that endpoint's page before you diff against your scraper.

The service is prepaid. When the balance runs out, calls return `insufficient_credits` (402) and nothing more is debited, and there is no automatic top-up. Read [how credits work](https://docs.perceptlayer.com/credits) before you point a job at it.

To see what one call returns, [get a free API key](https://perceptlayer.com/signup) and run an endpoint from your current scraper in the playground.

## Frequently asked questions

### Is a web scraper cheaper than an API?

It depends on volume and on how many hours upkeep takes. A scraper has no per-call fee, but you pay for selector fixes, proxies, servers and monitoring. Perceptlayer charges 1 credit per successful call, so 100,000 calls × 1 credit = 100,000 credits. Check [pricing](https://perceptlayer.com/pricing) for what those credits cost, then set that against your upkeep. Eight hours a month at an assumed $100 an hour is $800. Run the sum with your own figures.

### How often does a scraper break?

There's no fixed rate, because it depends on how often each site changes its markup. What matters is that breakage is unscheduled and often silent: the request succeeds but a selector returns nothing. Plan for it by checking that key fields are populated on every run and alerting when output volume drops.

### Does an API remove the need for retries?

No. You still retry transient failures. Perceptlayer returns `Retry-After` with the seconds to wait on 429 and 503 responses. Requests have no idempotency key, so a call whose reply was lost to a client timeout may have succeeded and been billed. Failed calls cost nothing and return `X-Credits-Used: 0`.

### Can I use both a scraper and an API?

Yes. Use the API for sources it covers and keep a scraper for gaps in coverage or fields it doesn't return. Keep the scraper narrow, and treat each remaining scraper as a cost you chose, with an owner and a reason.

### Are scraping and APIs both legal?

That depends on the site's terms, the data and your jurisdiction, so this isn't legal advice. Read the terms of any site you scrape and respect its robots rules. Some Perceptlayer endpoints return social, personal or SEO data and need the restricted-endpoint terms accepted in the dashboard first.

web scraping scraper vs api maintenance cost data engineering retries monitoring

## More from the blog

1.  [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 20 Oct 2026 Perceptlayer 6 min read
    
    ## [Detect Schema Changes in API JSON Responses](https://perceptlayer.com/blog/web-data-api-json-schema-changes)
    
    Infer a path-and-type schema from a recorded Google Search response (71 paths), diff each new response against it and alert on removed or retyped fields.
    
2.  [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 12 Oct 2026 Perceptlayer 7 min read
    
    ## [Incremental API Data Pulls: Fetch Only New Records](https://perceptlayer.com/blog/incremental-api-data-pulls)
    
    Pull only new listings: sort newest first, keep a high-water mark or a seen-ID set per query, and stop paging early. Python tested on Craigslist and eBay data.
    
3.  [Data engineering](https://perceptlayer.com/blog/category/data-engineering) 7 Oct 2026 Perceptlayer 10 min read
    
    ## [Scheduling Web Data Pulls: Cron, Queues or Workflows](https://perceptlayer.com/blog/schedule-web-data-pulls-cron-queue)
    
    Cron, a job queue or a workflow engine for recurring API pulls? Compare failure handling and credit cost, with a verdict per team size and a cron script.