Octocrawl
Web scraper for agents: blocked, empty and wrong pages reported as such, with an Evidence Record.
Connect it
Settings → Connectors → Add custom connector
URL: https://mcp.octocrawl.dev/mcpSettings → Apps & Connectors → Advanced → Developer mode
Create connector with URL: https://mcp.octocrawl.dev/mcp{
"mcpServers": {
"octocrawl": {
"url": "https://mcp.octocrawl.dev/mcp"
}
}
}claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp{
"servers": {
"octocrawl": {
"type": "http",
"url": "https://mcp.octocrawl.dev/mcp"
}
}
}{
"mcpServers": {
"octocrawl": {
"command": "npx",
"args": [
"-y",
"@octocrawl/mcp@0.6.0"
],
"env": {
"W2L_API_TOKEN": "<value>"
}
}
}
}claude mcp add octocrawl -- npx -y @octocrawl/mcp@0.6.02 tools, verified live
scraperead-onlyFetch one URL through the Octocrawl coverage ladder. Compact by default; set debug=true for the full audit. The result's warnings name what its content cannot vouch for: robots_overridden (robots.txt disallows the URL; a local server fetched it because you named it), or client_rendered_suspected when the HTTP page looks like a shell its scripts fill in and the browser rung found nothing better. It
mapread-onlyList a site's URLs without fetching each page: the start URL, the links on its page (read on the http lane alone; no browser) and the entries of the sitemaps the site declares (robots.txt Sitemap: lines, else /sitemap.xml), inside one deadline. Every URL is in the crawl's scope (the start host and its www twin, the start URL's path subtree, assets left out, similar URLs folded) and allowed by its
Change history
Alternatives in Browser & Scraping
Compare →SHShotsB
Create App Store screenshots, icons, ASO copy, localization, and revisions via hosted MCP.
DEDesignforyouB
Generate logos, social posts, app screenshots, comic panels & visual-novel assets from prompts.
SpytrendB
Analytics, advertising: Facebook/Meta, TikTok, competitor ads. Free plan; Spytrend OAuth in browser
SMScrapeless MCP ServerB
Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents.
WAWavedashB
Wavedash docs and workflow guidance for building and publishing browser games.
WSWeb Scraping MCP ServerB
Generic URL crawl + HTML extraction — fallback for sites without dedicated MCPs.