Blog / AI agents

MCP Server for Web Scraping: Tools Agents Can Call

2 Sept 2026Perceptlayer8 min read

An MCP server for web scraping exposes web access to an agent as named tools it can discover and call, instead of code you write and maintain per site. Perceptlayer's server at https://api.perceptlayer.com/mcp has three tools: search_endpoints, get_endpoint and call_endpoint. The agent finds an endpoint, looks it up, then calls it. A raw page scrape and a structured endpoint both cost 1 credit, so the real choice is the shape of the response your agent has to work with.

What does an MCP server do for a scraping agent?

An MCP server turns web access into a tool list the agent can discover at runtime. The Model Context Protocol is an open standard for connecting models to tools and data (Model Context Protocol specification, 2026). Without it, you hand-write a function per site and describe each one in the prompt. With it, the agent asks the server what exists and what each tool needs. If you're weighing this against defining your own functions, MCP vs Function Calling for Live Web Data in Agents compares the two.

For web data this matters because the catalogue is large: 248 websites and 1,659 endpoints, all returning JSON. You can't list that many tool definitions in a prompt. So the server doesn't try. It gives the agent a search tool over the catalogue and calls the right endpoint on demand.

The transport is Streamable HTTP, one of the two standard transports in the MCP transport specification (2026). That means a plain HTTPS URL, no local process to run.

Which tools does the Perceptlayer MCP server expose?

The server exposes three tools, and only one of them costs anything. search_endpoints finds endpoints in the catalogue, get_endpoint looks up one endpoint, and call_endpoint runs it. The first two are lookups and are free. The third takes an endpoint ID plus that endpoint's parameters, returns the endpoint's JSON response, and reports the credits used and remaining.

The agent's usual path is search, inspect, call. Because lookups are free, an agent can look at several candidate endpoints before it spends anything. If a call fails, you aren't billed for it.

What does the search, inspect, call sequence look like?

Take one task: an agent is asked for the top Google result for a query. It starts with search_endpoints, describing what it needs, and reads the candidates the catalogue returns. That step is free. It picks the Google Search endpoint that fits.

Next it calls get_endpoint on the chosen endpoint, which is also free, to look it up in the catalogue before spending anything. Last, it calls call_endpoint with the endpoint's ID and its parameters. For GET /v1/google-search/search that means a required query plus optional ones such as hl and safe. The call spends 1 credit if it succeeds and returns the endpoint's JSON, the same shape as the recorded response further down.

If the question had no good match in the catalogue, the first step would show that, and the agent could fall back to the scraper.

How do you add the server?

For Claude Code, one command registers it. Put your key in API_KEY first; you can get one free at /signup.

claude mcp add --transport http perceptlayer https://api.perceptlayer.com/mcp --header "Authorization: Bearer $API_KEY"

Other MCP clients need the same three things: the URL, the HTTP transport and the key in the Authorization header. The AI agents (MCP) docs are the place to check the details, and Claude MCP Web Search walks through a full search agent.

What does raw scraping give an agent?

Raw scraping takes a URL you choose and leaves the extraction to the agent. The Web Scraper API has one endpoint, GET /v1/web-scraper/scrape, at 1 credit per call. It takes a required url (up to 2048 characters) and three optional parameters: include_html, javascript and response_type.

Here's the request:

curl "https://api.perceptlayer.com/v1/web-scraper/scrape?url=https%3A%2F%2Fexample.com&include_html=0&javascript=0" \
  -H "X-API-Key: $API_KEY"

We haven't recorded a response for scrape, so this post can't show what it returns. The Web Scraper API page has a playground where you can run it against your own URL and see the response before you design the agent's extraction step.

The upside is reach: you pass the url, so it can point at a page no structured endpoint covers.

What does a structured endpoint give an agent instead?

A structured endpoint returns named fields for a known site, so the agent reads values rather than hunting for them. For Google results, GET /v1/google-search/search takes a required query and costs 1 credit, the same as a scrape. This is the recorded response to a search for best restaurants in Berlin with hl=en, trimmed:

curl "https://api.perceptlayer.com/v1/google-search/search?query=best+restaurants+in+Berlin&hl=en&safe=off" \
  -H "X-API-Key: $API_KEY"

{
  "search_information": {
    "query_displayed": "best restaurants in Berlin",
    "organic_results_count": 6
  },
  "organic_results": [
    {
      "position": 1,
      "title": "The Best Restaurants in Berlin - Food Guide & Map",
      "link": "https://andershusa.com/best-restaurants-berlin-germany-guide/",
      "redirect_link": "https://www.google.com/goto?url=CAESegHrOzAV7vOTOiRYGN3Rq_LJMgUCSUtmes2rLIk-rn2vHS6tGU7IIN38aZ1xjQeD3kZuDFfbW6QvUSum1Woyp6Hmrl5u9WEXLlEP2Ek60dzztg2yT13pxwC2ptSy…",
      "displayed_link": "https://andershusa.com › best-restaurants-berlin-german...",
      "source": "Anders & Kaitlin",
      "rank_group": 1,
      "rank_absolute": 4
    },
    …
  ],
  "related_searches": [
    {
      "query": "Best restaurants in Berlin for tourists",
      …
    }
  ],
  …
}

An agent asked for the top result's link reads organic_results[0].link. There's no page to parse. The same response also carries snippet text, related_searches and people_also_search_for, which an agent can use for follow-up queries. The Google Search API page lists all seven endpoints, and Google Search API: Get Search Results as JSON covers the parameters in depth.

One caution: the recorded excerpt includes long redirect_link values on each result. They arrive in the response, so a lean agent prompt should say which fields to use.

How do the two compare on credits and response shape?

On credits they tie: both calls cost 1. These are two different tasks, so it isn't a like-for-like comparison: one starts from a URL you supply, the other from a query. The table sets out what's documented.

What does a task cost in credits?

Credits add up per call, so work out the calls per task. Suppose an agent runs one search and scrapes three of the result pages: 1 + 3 = 4 credits per task. At 1,000 tasks that's 1,000 × 4 credits = 4,000 credits, which fits inside the Starter pack's 10,000 credits. See pricing for what credits cost. The free tier gives 1,000 credits a month after you verify an email address, no credit card.

The same task done with only structured calls might need one call, not four. That's a design choice, not a guarantee: it depends on whether the question needs page text at all. AI Agent API Cost Control shows how to cap this spend, and the credits docs list the rules.

When is raw scraping the right tool?

Raw scraping is right when no structured endpoint exists for the page, or when you need the page's own text. An agent reading a blog post, a changelog or a documentation page wants prose, not fields. The response_type parameter accepts json or markdown, so you can pick whichever your agent reads better. For what it costs to maintain your own scrapers instead, see Scraper or API: What Maintaining a Scraper Really Involves.

It's also the fallback for the long tail. The catalogue covers 248 websites; the web is far bigger. If the agent's question lands outside it, scrape is the endpoint to try. Check a site's terms and its robots.txt before you point an agent at it (RFC 9309, 2022).

Structured endpoints are right when the site is in the catalogue and the question is about a value: a ranking, a price, a rating. In that case search first. An agent that calls search_endpoints before scrape finds structured coverage when it exists, and the lookup costs nothing.

What happens when a call fails?

A failed call returns an HTTP status and a JSON body shaped as {"error": {"code", "message", "request_id"}}, and it costs nothing. The response carries X-Credits-Used: 0. Common codes are invalid_api_key (401), insufficient_credits (402), invalid_parameters (422) and rate_limited (429).

Two cases matter for agents. A 429 or 503 carries a Retry-After header with the seconds to wait, so the agent should sleep that long rather than retry in a loop. And because requests have no idempotency key, a request whose reply was lost to a client timeout may have succeeded and been billed. Retry Web API Calls Without Paying Twice covers that case, and the errors docs list every code.

The service is prepaid. When the balance reaches zero, calls return insufficient_credits (402) and nothing more is debited. Nothing tops up automatically.

What to do next

Add the server, then give the agent a question that has a catalogue answer and one that doesn't, and watch which tool it picks. To see the data before wiring anything, run a call in the playground on the Google Search API page, or get a free API key and try the cURL requests above. For a LangChain version of the same idea, see LangChain Custom Tool for Live Web Data. More agent guides are in the AI agents category.

mcpweb scrapingai agentstool callingcredits