--- title: "MCP Server for Web Scraping: Tools Agents Can Call" description: "An MCP server gives an agent three tools to find, inspect and call web endpoints. Use the scraper for uncatalogued pages, structured endpoints for the rest." url: https://perceptlayer.com/blog/mcp-server-for-web-scraping --- [Blog](https://perceptlayer.com/blog) / [AI agents](https://perceptlayer.com/blog/category/ai-agents) # MCP Server for Web Scraping: Tools Agents Can Call 2 Sept 2026 Perceptlayer 8 min read An MCP server for web scraping exposes web access to an agent as named tools it can discover and call, instead of code you write and maintain per site. Perceptlayer's server at `https://api.perceptlayer.com/mcp` has three tools: `search_endpoints`, `get_endpoint` and `call_endpoint`. The agent finds an endpoint, looks it up, then calls it. A raw page scrape and a structured endpoint both cost 1 credit, so the real choice is the shape of the response your agent has to work with. ## Key takeaways - The server has three tools. `search_endpoints` and `get_endpoint` only read the catalogue and cost nothing; `call_endpoint` spends credits, and only when the call succeeds. - The `/v1/web-scraper/scrape` endpoint takes a `url` (up to 2048 characters) and costs 1 credit. A structured endpoint such as `/v1/google-search/search` also costs 1 credit, and its recorded response has named fields such as `organic_results[0].link`. - Use the scraper for pages with no structured endpoint. Use a structured endpoint when the site is in the catalogue, because the agent reads documented fields. - Every successful response reports `X-Credits-Used` and `X-Credits-Remaining`, and failed calls return `X-Credits-Used: 0`, so an agent can track its own spend. ## What does an MCP server do for a scraping agent? An MCP server turns web access into a tool list the agent can discover at runtime. The Model Context Protocol is an open standard for connecting models to tools and data ( [Model Context Protocol specification](https://modelcontextprotocol.io/specification/2026-07-28), 2026). Without it, you hand-write a function per site and describe each one in the prompt. With it, the agent asks the server what exists and what each tool needs. If you're weighing this against defining your own functions, [MCP vs Function Calling for Live Web Data in Agents](https://perceptlayer.com/blog/mcp-vs-function-calling-web-data) compares the two. For web data this matters because the catalogue is large: 248 websites and 1,659 endpoints, all returning JSON. You can't list that many tool definitions in a prompt. So the server doesn't try. It gives the agent a search tool over the catalogue and calls the right endpoint on demand. The transport is Streamable HTTP, one of the two standard transports in the [MCP transport specification](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports) (2026). That means a plain HTTPS URL, no local process to run. ## Which tools does the Perceptlayer MCP server expose? The server exposes three tools, and only one of them costs anything. `search_endpoints` finds endpoints in the catalogue, `get_endpoint` looks up one endpoint, and `call_endpoint` runs it. The first two are lookups and are free. The third takes an endpoint ID plus that endpoint's parameters, returns the endpoint's JSON response, and reports the credits used and remaining. | Tool | What it does | Credits | | --- | --- | --- | | `search_endpoints` | Finds endpoints in the catalogue | 0 | | `get_endpoint` | Looks up one endpoint in the catalogue | 0 | | `call_endpoint` | Calls an endpoint by ID with its parameters | The endpoint's credits, on success only | The agent's usual path is search, inspect, call. Because lookups are free, an agent can look at several candidate endpoints before it spends anything. If a call fails, you aren't billed for it. ### What does the search, inspect, call sequence look like? Take one task: an agent is asked for the top Google result for a query. It starts with `search_endpoints`, describing what it needs, and reads the candidates the catalogue returns. That step is free. It picks the Google Search endpoint that fits. Next it calls `get_endpoint` on the chosen endpoint, which is also free, to look it up in the catalogue before spending anything. Last, it calls `call_endpoint` with the endpoint's ID and its parameters. For `GET /v1/google-search/search` that means a required `query` plus optional ones such as `hl` and `safe`. The call spends 1 credit if it succeeds and returns the endpoint's JSON, the same shape as the recorded response further down. If the question had no good match in the catalogue, the first step would show that, and the agent could fall back to the scraper. ### How do you add the server? For Claude Code, one command registers it. Put your key in `API_KEY` first; you can get one free at [/signup](https://perceptlayer.com/signup). ```bash claude mcp add --transport http perceptlayer https://api.perceptlayer.com/mcp --header "Authorization: Bearer $API_KEY" ``` Other MCP clients need the same three things: the URL, the HTTP transport and the key in the `Authorization` header. The [AI agents (MCP) docs](https://docs.perceptlayer.com/ai-agents) are the place to check the details, and [Claude MCP Web Search](https://perceptlayer.com/blog/claude-agent-live-search-mcp) walks through a full search agent. ## What does raw scraping give an agent? Raw scraping takes a URL you choose and leaves the extraction to the agent. The Web Scraper API has one endpoint, `GET /v1/web-scraper/scrape`, at 1 credit per call. It takes a required `url` (up to 2048 characters) and three optional parameters: `include_html`, `javascript` and `response_type`. | Parameter | Type | Notes | | --- | --- | --- | | `url` | string, required | Up to 2048 characters | | `include_html` | boolean, optional | See the [Scrape endpoint](https://perceptlayer.com/apis/web-scraper/scrape) | | `javascript` | boolean, optional | See the [Scrape endpoint](https://perceptlayer.com/apis/web-scraper/scrape) | | `response_type` | string, optional | `json` or `markdown` | Here's the request: ```bash curl "https://api.perceptlayer.com/v1/web-scraper/scrape?url=https%3A%2F%2Fexample.com&include_html=0&javascript=0" \ -H "X-API-Key: $API_KEY" ``` We haven't recorded a response for `scrape`, so this post can't show what it returns. The [Web Scraper API page](https://perceptlayer.com/apis/web-scraper) has a playground where you can run it against your own URL and see the response before you design the agent's extraction step. The upside is reach: you pass the `url`, so it can point at a page no structured endpoint covers. ## What does a structured endpoint give an agent instead? A structured endpoint returns named fields for a known site, so the agent reads values rather than hunting for them. For Google results, `GET /v1/google-search/search` takes a required `query` and costs 1 credit, the same as a scrape. This is the recorded response to a search for `best restaurants in Berlin` with `hl=en`, trimmed: ```bash curl "https://api.perceptlayer.com/v1/google-search/search?query=best+restaurants+in+Berlin&hl=en&safe=off" \ -H "X-API-Key: $API_KEY" ``` ```json { "search_information": { "query_displayed": "best restaurants in Berlin", "organic_results_count": 6 }, "organic_results": [ { "position": 1, "title": "The Best Restaurants in Berlin - Food Guide & Map", "link": "https://andershusa.com/best-restaurants-berlin-germany-guide/", "redirect_link": "https://www.google.com/goto?url=CAESegHrOzAV7vOTOiRYGN3Rq_LJMgUCSUtmes2rLIk-rn2vHS6tGU7IIN38aZ1xjQeD3kZuDFfbW6QvUSum1Woyp6Hmrl5u9WEXLlEP2Ek60dzztg2yT13pxwC2ptSy…", "displayed_link": "https://andershusa.com › best-restaurants-berlin-german...", "source": "Anders & Kaitlin", "rank_group": 1, "rank_absolute": 4 }, … ], "related_searches": [ { "query": "Best restaurants in Berlin for tourists", … } ], … } ``` An agent asked for the top result's link reads `organic_results[0].link`. There's no page to parse. The same response also carries `snippet` text, `related_searches` and `people_also_search_for`, which an agent can use for follow-up queries. The [Google Search API page](https://perceptlayer.com/apis/google-search) lists all seven endpoints, and [Google Search API: Get Search Results as JSON](https://perceptlayer.com/blog/google-search-api-json) covers the parameters in depth. One caution: the recorded excerpt includes long `redirect_link` values on each result. They arrive in the response, so a lean agent prompt should say which fields to use. ## How do the two compare on credits and response shape? On credits they tie: both calls cost 1. These are two different tasks, so it isn't a like-for-like comparison: one starts from a URL you supply, the other from a query. The table sets out what's documented. | | Web Scraper `scrape` | Structured endpoint ( `search`) | | --- | --- | --- | | Credits per call | 1 | 1 | | Input | `url`, required | `query`, required | | Response | `json` or `markdown` ( `response_type`) | Named fields such as `organic_results[0].link` | | Sites covered | The URL you pass | Sites in the catalogue | ### What does a task cost in credits? Credits add up per call, so work out the calls per task. Suppose an agent runs one `search` and scrapes three of the result pages: 1 + 3 = 4 credits per task. At 1,000 tasks that's 1,000 × 4 credits = 4,000 credits, which fits inside the Starter pack's 10,000 credits. See [pricing](https://perceptlayer.com/pricing) for what credits cost. The free tier gives 1,000 credits a month after you verify an email address, no credit card. The same task done with only structured calls might need one call, not four. That's a design choice, not a guarantee: it depends on whether the question needs page text at all. [AI Agent API Cost Control](https://perceptlayer.com/blog/ai-agent-web-data-costs) shows how to cap this spend, and the [credits docs](https://docs.perceptlayer.com/credits) list the rules. ## When is raw scraping the right tool? Raw scraping is right when no structured endpoint exists for the page, or when you need the page's own text. An agent reading a blog post, a changelog or a documentation page wants prose, not fields. The `response_type` parameter accepts `json` or `markdown`, so you can pick whichever your agent reads better. For what it costs to maintain your own scrapers instead, see [Scraper or API: What Maintaining a Scraper Really Involves](https://perceptlayer.com/blog/scraper-vs-api-maintenance-cost). It's also the fallback for the long tail. The catalogue covers 248 websites; the web is far bigger. If the agent's question lands outside it, `scrape` is the endpoint to try. Check a site's terms and its `robots.txt` before you point an agent at it ( [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), 2022). Structured endpoints are right when the site is in the catalogue and the question is about a value: a ranking, a price, a rating. In that case search first. An agent that calls `search_endpoints` before `scrape` finds structured coverage when it exists, and the lookup costs nothing. ## What happens when a call fails? A failed call returns an HTTP status and a JSON body shaped as `{"error": {"code", "message", "request_id"}}`, and it costs nothing. The response carries `X-Credits-Used: 0`. Common codes are `invalid_api_key` (401), `insufficient_credits` (402), `invalid_parameters` (422) and `rate_limited` (429). Two cases matter for agents. A 429 or 503 carries a `Retry-After` header with the seconds to wait, so the agent should sleep that long rather than retry in a loop. And because requests have no idempotency key, a request whose reply was lost to a client timeout may have succeeded and been billed. [Retry Web API Calls Without Paying Twice](https://perceptlayer.com/blog/retry-web-api-calls-without-paying-twice) covers that case, and the [errors docs](https://docs.perceptlayer.com/errors) list every code. The service is prepaid. When the balance reaches zero, calls return `insufficient_credits` (402) and nothing more is debited. Nothing tops up automatically. ## What to do next Add the server, then give the agent a question that has a catalogue answer and one that doesn't, and watch which tool it picks. To see the data before wiring anything, run a call in the playground on the [Google Search API page](https://perceptlayer.com/apis/google-search), or [get a free API key](https://perceptlayer.com/signup) and try the cURL requests above. For a LangChain version of the same idea, see [LangChain Custom Tool for Live Web Data](https://perceptlayer.com/blog/langchain-custom-tool-web-data). More agent guides are in the [AI agents category](https://perceptlayer.com/blog/category/ai-agents). ## APIs in this post - [Web Scraper **1 credit per call** · 1 endpoint](https://perceptlayer.com/apis/web-scraper) - [Google Search **1 credit per call** · 7 endpoints](https://perceptlayer.com/apis/google-search) ## Frequently asked questions ### Do the MCP lookup tools cost credits? No. `search_endpoints` and `get_endpoint` only look up the catalogue and cost nothing. Only `call_endpoint` spends credits, it spends the endpoint's listed credits, and it does so only when the call succeeds. Most endpoints cost 1 credit. Failed calls cost nothing and return `X-Credits-Used: 0`. ### How does the agent authenticate with the MCP server? Claude Code sends your API key in the `Authorization: Bearer` header, as in the command above. Direct HTTP calls use the `X-API-Key` header instead. Keys start with `sk_live_`. See [authentication](https://docs.perceptlayer.com/authentication) for the API key details. ### Is a scrape cheaper than a structured call? No. The `scrape` endpoint costs 1 credit and so does `search`. A structured call costs the credits shown on its endpoint, and nearly all cost 1. So the decision rests on the input and the response shape, not on price per call. ### How much does it cost to run an agent for a month? Multiply calls per task by tasks by 1 credit. An agent making 10,000 calls a month uses 10,000 calls × 1 credit = 10,000 credits, which is the Starter pack's 10,000 credits. Purchased credits last 12 months, and there's no subscription. See [pricing](https://perceptlayer.com/pricing) for what credits cost and for the Build and Growth packs. ### What happens if my agent hits the rate limit? A rate-limited response returns 429 with a `Retry-After` header giving the seconds to wait. Temporarily unavailable responses (503) carry the same header. Have the agent wait that long before retrying, and cap the number of retries so a stuck loop can't run indefinitely. The [rate limits docs](https://docs.perceptlayer.com/rate-limits) have details. mcp web scraping ai agents tool calling credits ## More from the blog 1. [AI agents](https://perceptlayer.com/blog/category/ai-agents) 12 Oct 2026 Perceptlayer 7 min read ## [MCP Server Too Many Tools: Keeping Agent Context Small](https://perceptlayer.com/blog/mcp-too-many-tools-context) One tool per endpoint for 1,659 endpoints is about 567,000 characters of schema. A 10-step checklist to keep an MCP server's tool list small. 2. [AI agents](https://perceptlayer.com/blog/category/ai-agents) 16 Sept 2026 Perceptlayer 7 min read ## [MCP Tool Call Error Handling: When an Agent Should Retry](https://perceptlayer.com/blog/mcp-tool-call-errors-agent-retries) Each call\_endpoint error code mapped to one agent action (fix, wait, stop or ask the user), with the exact error text the model sees and a system prompt. 3. [AI agents](https://perceptlayer.com/blog/category/ai-agents) 24 Jul 2026 Perceptlayer 7 min read ## [LLM Tool Schema Design: A Checklist for Web APIs](https://perceptlayer.com/blog/agent-tool-schema-design-web-apis) LLM tool schema design checklist, applied to the Google Maps simple-search endpoint: a before and after schema that cuts 28 result fields to 7.