Web Scraping MCP Server
Generic URL crawl + HTML extraction — fallback for sites without dedicated MCPs.
Connect it
Settings → Connectors → Add custom connector
URL: https://nexgendata-mcp-proxy.steve-corbeil.workers.dev/web-scraping-mcp-server/mcpSettings → Apps & Connectors → Advanced → Developer mode
Create connector with URL: https://nexgendata-mcp-proxy.steve-corbeil.workers.dev/web-scraping-mcp-server/mcp{
"mcpServers": {
"web-scraping-mcp-server": {
"url": "https://nexgendata-mcp-proxy.steve-corbeil.workers.dev/web-scraping-mcp-server/mcp",
"headers": {
"Authorization": "<YOUR_AUTHORIZATION>"
}
}
}
}claude mcp add --transport http web-scraping-mcp-server https://nexgendata-mcp-proxy.steve-corbeil.workers.dev/web-scraping-mcp-server/mcp --header "Authorization: <value>"{
"servers": {
"web-scraping-mcp-server": {
"type": "http",
"url": "https://nexgendata-mcp-proxy.steve-corbeil.workers.dev/web-scraping-mcp-server/mcp",
"headers": {
"Authorization": "<YOUR_AUTHORIZATION>"
}
}
}
}3 tools, verified live
crawl_websiteread-onlyCrawl a website and extract its content as structured data. Args: url: Website URL to crawl (e.g. 'https://example.com') max_pages: Max pages to crawl (default 5)
google_searchread-onlySearch Google and return organic results with titles, URLs, snippets. Args: query: Search query (e.g. 'best python libraries 2026') max_results: Max results (default 10)
wikipedia_lookupread-onlyLook up a Wikipedia article and return its content. Args: topic: Topic to look up (e.g. 'Artificial intelligence')
Alternatives in Browser & Scraping
Compare →GAGatewayB
Durable self for AI agents: one-call resume, memory, real browser, free chat + hire real humans.
ZMZenrows MCPB
Zenrows MCP server — Fetch, Extract, Batch, and Browser Sessions for AI coding assistants
UNUnbrowseB
Websites as APIs: read pages, run site tasks via learned APIs, cloud browser with saved logins.
CMCrawlora MCPB
Hosted MCP: 3631 structured web-data tools across 471 platform groups.
FAfastCRWB
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
SMScrapeunblocker MCP RemoteB
Fetch any page's HTML or AI-parsed JSON, Google results, or 100+ site plugins via ScrapeUnblocker