Ai Crawler Robots Monitor MCP server
{
"mcpServers": {
"actablesite-ai-crawler-robots-monitor": {
"type": "http",
"url": "https://actablesite.com/mcp/"
}
}
}{
"mcpServers": {
"actablesite-ai-crawler-robots-monitor": {
"type": "http",
"url": "https://actablesite.com/mcp/"
}
}
}{
"mcpServers": {
"actablesite-ai-crawler-robots-monitor": {
"type": "http",
"url": "https://actablesite.com/mcp/"
}
}
}{
"servers": {
"actablesite-ai-crawler-robots-monitor": {
"type": "http",
"url": "https://actablesite.com/mcp/"
}
}
}{
"mcpServers": {
"actablesite-ai-crawler-robots-monitor": {
"type": "http",
"url": "https://actablesite.com/mcp/"
}
}
}file .mcp.json (in your project root)file claude_desktop_config.jsonfile ~/.cursor/mcp.jsonfile .vscode/mcp.jsonfile ~/.codeium/windsurf/mcp_config.jsonversion 1.4.2
What it does
This server audits public websites for AI crawler policies, checks compliance with OpenAI, Anthropic, Perplexity, and Google crawler tokens, and monitors changes to robots.txt, sitemap.xml, and llms.txt files. It is for developers and website operators who need to verify and track how AI systems can access their public sites.
- Audit a public website to check its identity, offer evidence, discovery files like robots.txt and sitemap.xml, and structured metadata.
- Evaluate whether a website's AI crawler policy allows or blocks specific AI providers including OpenAI, Anthropic, Perplexity, and Google.
- Review pricing and features of Crawler Watch (automated 15-minute monitoring) and Practice Radar (weekly crawler activity reports) services.
Tools this server exposes
| Tool | Does |
|---|---|
| audit_public_website | Checks a public website's identity, offer evidence, action paths, discovery files, metadata, and structured data. |
| check_ai_crawler_policy | Evaluates AI crawler policy compliance for OpenAI, Anthropic, Perplexity, and Google crawler tokens. |
| get_crawler_watch_offer | Returns metadata for the Crawler Watch service: $9 monthly price, monitoring scope, 15-minute check cadence, and limitations. |
| get_full_report_offer | Returns metadata for an optional $19 one-time detailed report. |
| get_practice_radar_offer | Returns metadata for the Practice Radar service: $39 monthly price, weekly editions, delivery details, and material limitations. |
Before you enable it
An MCP server runs with whatever access you give it. Start with a development account rather than production, and check which tools the server exposes before you let an agent call them unattended.