ai.txt is an emerging IETF standard that allows website owners to communicate machine-readable policies to AI crawlers and LLM agents.
How ai.txt differs from robots.txt
| Feature | robots.txt | ai.txt |
|---|---|---|
| Purpose | Control crawling access | Control use of content (training, retrieval, citation) |
| Standard | Long-standing (1994) | Emerging IETF draft (2025) |
| Supported by | All major crawlers | GPTBot, ClaudeBot, Google-Extended (partial) |
| Granularity | Path-level | Use-case level (train, retrieve, cite) |
Basic ai.txt template
Create a file at /ai.txt on your website root:
# ai.txt - AI Crawler Policies for example.com
# Allow all AI crawlers for retrieval (real-time answers)
User-agent: *
Retrieve: allow
# Block training on my content
Train: disallow
# Allow citation with attribution
Cite: allow
# Block specific paths from AI retrieval
Disallow: /private/Directives explained
| Directive | Meaning | Values |
|---|---|---|
Train | Use content to train/fine-tune AI models | allow, disallow |
Retrieve | Use content in real-time retrieval (RAG) | allow, disallow |
Cite | Include citations/sources | allow, disallow, require-attribution |
Disallow | Paths AI crawlers cannot access | URL paths |
How to link ai.txt from HTML
Add to of your pages:
<link rel="ai-text" href="/ai.txt" type="text/plain">Will AI crawlers respect ai.txt?
- GPTBot has announced support for ai.txt (as of 2026)
- Google-Extended has partial support (Training directives)
- ClaudeBot supports the
Traindirective - PerplexityBot supports
RetrieveandCite
Not all crawlers support all directives yet, but the standard is rapidly being adopted.
Check your AI SEO Score — free, instant, no sign-up required
Get My Free AI SEO Score → View Full Audit Report