AIReady
Scan and fix your site's AI discoverability: crawler access, llms.txt, JSON-LD. Free.
Connect it
Settings → Connectors → Add custom connector
URL: https://ai-ready.magicteams.ai/mcpSettings → Apps & Connectors → Advanced → Developer mode
Create connector with URL: https://ai-ready.magicteams.ai/mcp{
"mcpServers": {
"ai-ready": {
"url": "https://ai-ready.magicteams.ai/mcp"
}
}
}claude mcp add --transport http ai-ready https://ai-ready.magicteams.ai/mcp{
"servers": {
"ai-ready": {
"type": "http",
"url": "https://ai-ready.magicteams.ai/mcp"
}
}
}5 tools, verified live
check_ai_readinessread-onlyLive scan of one domain's AI discoverability: fetches its /robots.txt, /llms.txt, /sitemap.xml and homepage, then returns a 0-100 score, grade, per-crawler allow/block status and prioritized fixes. Use to audit a real site. Do NOT use for general advice on which bots to allow (use crawler_policy_guide) or to write new rules (use robots_txt_for_ai).
crawler_policy_guideread-onlyStatic reference of 13 known AI crawlers and control tokens (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended...): owner, purpose, whether each drives AI citations or training, plus rules of thumb. Use to decide which bots to allow or block. No site is fetched. Do NOT use to audit a real site (use check_ai_readiness) or to produce rules (use robots_txt_for_ai).
robots_txt_for_airead-onlyGenerate robots.txt User-agent rules for all known AI crawlers from a chosen policy (max_visibility, search_only, block_training, block_all_ai), with optional Content-Signal and Sitemap lines, to paste into an existing robots.txt. Use when writing AI crawler rules. Do NOT use to check a live site's current rules (use check_ai_readiness) or for bot background (use crawler_policy_guide).
llms_txt_draftread-onlyDraft a starter llms.txt (Markdown: # name, > one-line summary, key page links, contact) from business details, to publish at the site root so AI agents get a curated summary. Use when a site has no llms.txt. Do NOT use for crawler access rules (use robots_txt_for_ai) or schema.org markup (use schema_jsonld_sample).
schema_jsonld_sampleread-onlyGenerate a schema.org JSON-LD snippet (Organization, LocalBusiness, Product with Offer, or FAQPage) from your details, with placement tips, so search engines and AI assistants can identify the entity. Use when adding structured data to a page. Do NOT use to check whether a site already has structured data (use check_ai_readiness) or for llms.txt (use llms_txt_draft).
Alternatives in Browser & Scraping
Compare →
Writ CloudB
Read, crawl and act on websites, signed in as the user, and turn any site into an API.
SHShadowCrawlB
Stealth scraping & search. Bypasses Cloudflare, DataDome & LinkedIn via Cyborg HITL approach.
OBOya BrowserB
Real Chrome for agents: start a browser, read pages as numbered markdown, click, type, hand off.
BRBrowserlessB
Official Browserless MCP server: browser automation, scraping, screenshots and PDFs for AI agents.
BRBrowserstackB
BrowserStack's Official MCP Server
SupadataB
Video transcripts, metadata and structured data from YouTube, TikTok, Instagram, X; web scraping.