Blog / Data engineering
Data engineering
Building reliable pipelines on web data: scrapers versus APIs, rate limits, retries, data quality, storage and cost control.
Detect Schema Changes in API JSON Responses
Infer a path-and-type schema from a recorded Google Search response (71 paths), diff each new response against it and alert on removed or retyped fields.
Concurrent API Requests in Python: Asyncio Under Rate Limits
A Python asyncio fetcher that paces calls to your per-second limit, honours Retry-After and prices a 20,000-call batch at 20,000 credits.
Incremental API Data Pulls: Fetch Only New Records
Pull only new listings: sort newest first, keep a high-water mark or a seen-ID set per query, and stop paging early. Python tested on Craigslist and eBay data.
Scheduling Web Data Pulls: Cron, Queues or Workflows
Cron, a job queue or a workflow engine for recurring API pulls? Compare failure handling and credit cost, with a verdict per team size and a cron script.
Save API JSON to Parquet in Python
Write API JSON to date-partitioned Parquet with pyarrow and a fixed schema, so a price that arrives as 441 one day and 393.73 the next doesn't break it.
Handle Web Data API Errors in Python: 4xx, 5xx and Timeouts
Map the API's error codes to retry, fix or stop in Python: 429 and 503 honour Retry-After, 4xx stops, and failed calls cost 0 credits.
Web Data Quality Checks for Scraped and API Data
Seven web data quality checks in Python, with code written against a recorded Zillow search response: schema drift, null rates, duplicate zpids and outliers.
Cache API Responses in Python: TTLs and Credit Savings
A SQLite cache for API responses, keyed on endpoint and parameters, with per-endpoint TTLs and the credits each cache hit saves.
Web Data to a Warehouse: Loading API JSON into Postgres
Land AutoScout24 search JSON in a Postgres JSONB table, flatten it with a view, then upsert on listing_id. Code and credit cost per refresh.
Scraper or API: What Maintaining a Scraper Really Involves
What a scraper costs to keep running: selector fixes, proxies, retries and monitoring, set against an API at 1 credit per successful call.
Normalise Prices Across Currencies and Sources in Python
Parse '162,90 €' and $89.99 into Decimals, tag each price with its currency, convert with a dated ECB reference rate, and keep the original value for audit.
Paginate a Web Data API in Python: Safe Page Loops
Page through the eBay search endpoint in Python with has_more, dedupe by product_id and a credit ceiling. Each page costs 1 credit.
Estimate a Web Data Pipeline's Monthly Credit Cost
A 10-step checklist that turns a data requirement into calls a month (entities, refresh rate, pages, fan-out, margin), with worked arithmetic against packs.
Flatten Nested JSON to CSV in Python: Two Ways Compared
Flatten a recorded Zillow response to CSV with pandas json_normalize (60 columns) and plain Python (63), split photo lists into a child table, fix integers.
Retry Web API Calls Without Paying Twice
Retry only 429, 502 and 503. Failed responses return X-Credits-Used: 0, but a timeout is the gap. The credit arithmetic, a Python loop and a balance check.
Deduplicate Listings Across Sources: Matching Without IDs
Match car listings from AutoScout24 and AutoTrader UK with no shared ID: normalise make and model, block, then score year, price and mileage in Python.