Crawlie MCP server
{
"mcpServers": {
"crawlie": {
"type": "http",
"url": "https://crawlie.app/mcp"
}
}
}{
"mcpServers": {
"crawlie": {
"type": "http",
"url": "https://crawlie.app/mcp"
}
}
}{
"mcpServers": {
"crawlie": {
"type": "http",
"url": "https://crawlie.app/mcp"
}
}
}{
"servers": {
"crawlie": {
"type": "http",
"url": "https://crawlie.app/mcp"
}
}
}{
"mcpServers": {
"crawlie": {
"type": "http",
"url": "https://crawlie.app/mcp"
}
}
}file .mcp.json (in your project root)file claude_desktop_config.jsonfile ~/.cursor/mcp.jsonfile .vscode/mcp.jsonfile ~/.codeium/windsurf/mcp_config.jsonversion 0.1.0
What it does
Crawlie is a technical SEO and generative-engine readiness crawler that audits websites for broken links, redirects, missing metadata, and 40+ SEO-related checks. It runs locally as a CLI tool and MCP server, or as a hosted cloud service, providing plain-English guidance on fixes and reporting scores for health, AI-search readiness, and accessibility.
- Audit a single webpage or entire site for broken links, redirects, missing metadata, and SEO issues before deployment.
- Generate shareable HTML reports or JSON output to integrate site-health data into CI/CD pipelines or scripting workflows.
- Let an LLM agent run a full audit and act on findings through the MCP server without manual intervention.
Tools this server exposes
| Tool | Does |
|---|---|
| crawl_site | Crawls a website respecting robots.txt and sitemap.xml, checking for 40+ SEO and generative-engine issues. |
| audit_url | Audits one or more specific URLs for technical SEO, accessibility, and AI-search readiness. |
| top_fixes | Returns the highest-impact fixes for a crawl report. |
| geo_gaps | Identifies gaps in generative-engine optimization readiness. |
| affected_urls | Lists URLs affected by a specific issue type. |
| diff_reports | Compares two crawl reports to show changes. |
| crawl_status | Polls the status of a long-running crawl in progress. |
| get_report | Retrieves a previously saved cloud crawl report. |
| list_reports | Lists saved cloud crawl reports for a team. |
Before you enable it
An MCP server runs with whatever access you give it. Start with a development account rather than production, and check which tools the server exposes before you let an agent call them unattended.