Even with perfect robots.txt and meta tags, slow servers can block AI crawlers more effectively than any Disallow rule.
How AI crawlers behave differently
| Crawler | Timeout | Rate limit respect |
|---|---|---|
| GPTBot | ~3 seconds | Yes (Crawl-delay) |
| ClaudeBot | ~2 seconds | Yes |
| Google-Extended | ~5 seconds | Yes (via Search Console) |
| PerplexityBot | ~2 seconds | Yes |
If your server takes longer than these thresholds, the crawler abandons the request and may mark your site as "slow" — reducing future crawl frequency.
Critical metric: TTFB (Time To First Byte)
TTFB measures how quickly your server responds to a request. For AI crawlers:
- Excellent: < 200ms
- Acceptable: 200-500ms
- Too slow: > 800ms (risks timeout)
The Retry-After header
You can use the Retry-After HTTP header to tell AI crawlers when to come back:
HTTP/1.1 429 Too Many Requests
Retry-After: 3600This tells the bot to wait 1 hour before trying again. Useful for:
- Managing server load
- Throttling aggressive crawlers
- Communicating maintenance windows
How to optimise for AI crawlers
| Fix | Impact on TTFB | Difficulty |
|---|---|---|
| Use CDN (Cloudflare, AWS CloudFront) | 50-80% reduction | Low |
| Enable caching (Redis, Varnish) | 90%+ reduction after first hit | Medium |
| Optimise database queries | Varies | Medium-High |
| Upgrade hosting (dedicated/VPS) | 30-60% improvement | Low (cost) |
If AI crawlers are overwhelming your server
- Use
Crawl-delay: 1in robots.txt (seconds) - Implement
Retry-Afterheaders for 429 responses - Upgrade server resources
Check your AI SEO Score — free, instant, no sign-up required
Get My Free AI SEO Score → View Full Audit Report