---
title: "Data engineering: articles | Perceptlayer"
description: "Building reliable pipelines on web data: scrapers versus APIs, rate limits, retries, data quality, storage and cost control."
url: https://perceptlayer.com/blog/category/data-engineering
---

[Blog](https://perceptlayer.com/blog) / Data engineering

# Data engineering

Building reliable pipelines on web data: scrapers versus APIs, rate limits, retries, data quality, storage and cost control.

1.  20 Oct 2026 Perceptlayer 6 min read
    
    ## [Detect Schema Changes in API JSON Responses](https://perceptlayer.com/blog/web-data-api-json-schema-changes)
    
    Infer a path-and-type schema from a recorded Google Search response (71 paths), diff each new response against it and alert on removed or retyped fields.
    
2.  19 Oct 2026 Perceptlayer 7 min read
    
    ## [Concurrent API Requests in Python: Asyncio Under Rate Limits](https://perceptlayer.com/blog/concurrent-api-requests-python-rate-limit)
    
    A Python asyncio fetcher that paces calls to your per-second limit, honours Retry-After and prices a 20,000-call batch at 20,000 credits.
    
3.  12 Oct 2026 Perceptlayer 7 min read
    
    ## [Incremental API Data Pulls: Fetch Only New Records](https://perceptlayer.com/blog/incremental-api-data-pulls)
    
    Pull only new listings: sort newest first, keep a high-water mark or a seen-ID set per query, and stop paging early. Python tested on Craigslist and eBay data.
    
4.  7 Oct 2026 Perceptlayer 10 min read
    
    ## [Scheduling Web Data Pulls: Cron, Queues or Workflows](https://perceptlayer.com/blog/schedule-web-data-pulls-cron-queue)
    
    Cron, a job queue or a workflow engine for recurring API pulls? Compare failure handling and credit cost, with a verdict per team size and a cron script.
    
5.  29 Sept 2026 Perceptlayer 6 min read
    
    ## [Save API JSON to Parquet in Python](https://perceptlayer.com/blog/save-api-json-to-parquet-python)
    
    Write API JSON to date-partitioned Parquet with pyarrow and a fixed schema, so a price that arrives as 441 one day and 393.73 the next doesn't break it.
    
6.  22 Sept 2026 Perceptlayer 7 min read
    
    ## [Handle Web Data API Errors in Python: 4xx, 5xx and Timeouts](https://perceptlayer.com/blog/handle-web-data-api-errors-python)
    
    Map the API's error codes to retry, fix or stop in Python: 429 and 503 honour Retry-After, 4xx stops, and failed calls cost 0 credits.
    
7.  11 Sept 2026 Perceptlayer 7 min read
    
    ## [Web Data Quality Checks for Scraped and API Data](https://perceptlayer.com/blog/web-data-quality-monitoring-checklist)
    
    Seven web data quality checks in Python, with code written against a recorded Zillow search response: schema drift, null rates, duplicate zpids and outliers.
    
8.  22 Aug 2026 Perceptlayer 10 min read
    
    ## [Cache API Responses in Python: TTLs and Credit Savings](https://perceptlayer.com/blog/cache-web-data-api-responses)
    
    A SQLite cache for API responses, keyed on endpoint and parameters, with per-endpoint TTLs and the credits each cache hit saves.
    
9.  19 Aug 2026 Perceptlayer 7 min read
    
    ## [Web Data to a Warehouse: Loading API JSON into Postgres](https://perceptlayer.com/blog/load-web-data-json-into-postgres)
    
    Land AutoScout24 search JSON in a Postgres JSONB table, flatten it with a view, then upsert on listing\_id. Code and credit cost per refresh.
    
10.  15 Aug 2026 Perceptlayer 11 min read
     
     ## [Scraper or API: What Maintaining a Scraper Really Involves](https://perceptlayer.com/blog/scraper-vs-api-maintenance-cost)
     
     What a scraper costs to keep running: selector fixes, proxies, retries and monitoring, set against an API at 1 credit per successful call.
     
11.  14 Aug 2026 Perceptlayer 7 min read
     
     ## [Normalise Prices Across Currencies and Sources in Python](https://perceptlayer.com/blog/normalize-prices-multiple-currencies)
     
     Parse '162,90 €' and $89.99 into Decimals, tag each price with its currency, convert with a dated ECB reference rate, and keep the original value for audit.
     
12.  6 Aug 2026 Perceptlayer 8 min read
     
     ## [Paginate a Web Data API in Python: Safe Page Loops](https://perceptlayer.com/blog/paginate-web-data-api-python)
     
     Page through the eBay search endpoint in Python with has\_more, dedupe by product\_id and a credit ceiling. Each page costs 1 credit.
     
13.  4 Aug 2026 Perceptlayer 6 min read
     
     ## [Estimate a Web Data Pipeline's Monthly Credit Cost](https://perceptlayer.com/blog/web-data-pipeline-cost-estimate)
     
     A 10-step checklist that turns a data requirement into calls a month (entities, refresh rate, pages, fan-out, margin), with worked arithmetic against packs.
     
14.  31 Jul 2026 Perceptlayer 6 min read
     
     ## [Flatten Nested JSON to CSV in Python: Two Ways Compared](https://perceptlayer.com/blog/web-data-to-csv-python)
     
     Flatten a recorded Zillow response to CSV with pandas json\_normalize (60 columns) and plain Python (63), split photo lists into a child table, fix integers.
     
15.  31 Jul 2026 Perceptlayer 10 min read
     
     ## [Retry Web API Calls Without Paying Twice](https://perceptlayer.com/blog/retry-web-api-calls-without-paying-twice)
     
     Retry only 429, 502 and 503. Failed responses return X-Credits-Used: 0, but a timeout is the gap. The credit arithmetic, a Python loop and a balance check.
     
16.  28 Jul 2026 Perceptlayer 10 min read
     
     ## [Deduplicate Listings Across Sources: Matching Without IDs](https://perceptlayer.com/blog/deduplicate-listings-across-sources)
     
     Match car listings from AutoScout24 and AutoTrader UK with no shared ID: normalise make and model, block, then score year, price and mileage in Python.