How to check if AI bots can crawl your website
AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) need explicit permission to access your content. Here's how to check:
Method 1: Manual inspection
- Check robots.txt — Go to
https://yourdomain.com/robots.txt. Look for lines like:
- User-agent: GPTBot followed by Disallow: / (blocked) or Allow: / (allowed)
- User-agent: ClaudeBot
- User-agent: Google-Extended
- Check meta tags — View page source and search for:
- (blocks AI)
-
- Check for ai.txt — Try
https://yourdomain.com/ai.txtor/llms.txt(emerging standards).
Method 2: Automated tool (recommended)
UltraScout's free AI SEO Score does all this in 10 seconds:
- Parses your robots.txt for all major AI user-agents
- Detects AI-blocking meta directives
- Checks for ai.txt and llms.txt presence
- Provides a crawlability score from 0-100
What the results mean
| Score | Meaning |
|---|---|
| 80-100 | Fully accessible — AI crawlers can read your content |
| 50-79 | Partial access — some bots blocked or missing directives |
| 0-49 | Poor — critical blocks found |
No signup required. Instant results.
Check your AI SEO Score — free, instant, no sign-up required
Get My Free AI SEO Score → View Full Audit Report