Quick comparison
| Feature | ClaudeBot | Google-Extended |
|---|---|---|
| Owner | Anthropic | |
| Purpose | Real-time reading for Claude AI answers | Training data for Google's AI models (Gemini, AI Overviews) |
| What it affects | Responses in Claude.ai and Claude API | Google AI Overviews, Gemini answers |
| Default behaviour | Allowed unless blocked | Allowed unless blocked |
| Scope | Single AI assistant | Entire Google ecosystem |
Google-Extended explained
Google-Extended is a user-agent token introduced in September 2023. It gives website owners control over whether Google can use their content to train Gemini and future generative AI models.
What Google-Extended does NOT affect:
- Regular Google Search ranking
- Crawling for traditional search results (Googlebot)
What it DOES affect:
- Training data for Google's LLMs (Gemini)
- Future generative AI products
- Some aspects of SGE (Search Generative Experience)
ClaudeBot explained
ClaudeBot is Anthropic's crawler that reads your website to answer questions in Claude AI (similar to ChatGPT).
Key differences from Google-Extended:
- ClaudeBot reads your site in real-time for each answer
- Not just training — it's live retrieval
- Blocking ClaudeBot immediately removes you from Claude answers
Which one matters more?
| If you care about... | Priority |
|---|---|
| Google AI Overviews appearing in search results | Google-Extended |
| Being cited in Gemini answers | Google-Extended |
| Being cited in Claude AI answers | ClaudeBot |
| Global reach (billions of searches daily) | Google-Extended |
| Technical audiences (Claude popular among developers) | ClaudeBot |
Verdict: For most businesses, Google-Extended has a larger potential reach because Google AI Overviews appear in 16% of desktop searches. However, Claude is growing rapidly.
How to allow both (recommended)
Add to your robots.txt:
User-agent: Google-Extended
Allow: /
User-agent: ClaudeBot
Allow: /The bottom line
- Don't block either unless you have a specific legal or policy reason
- Both crawlers respect robots.txt and rate limits
- Allowing them costs nothing and can improve your AI visibility
Check your AI SEO Score — free, instant, no sign-up required
Get My Free AI SEO Score → View Full Audit Report