UltraScout AI › Learn › How to block GPTBot using robots.txt?

How to Block GPTBot Using robots.txt

GPTBot is OpenAI's web crawler that collects data for ChatGPT and other models. Here's how to control its access.

Why block GPTBot?

Common reasons to block:

• Content sensitivity — You don't want proprietary info in ChatGPT
• Server load — High crawl volumes (though GPTBot respects rate limits)
• Policy compliance — Legal or contractual restrictions on AI training

How to block GPTBot (complete block)

Add these lines to your robots.txt file:

User-agent: GPTBot
Disallow: /

How to partially block GPTBot (specific directories)

User-agent: GPTBot
Disallow: /private/
Disallow: /internal/
Allow: /

How to allow GPTBot (recommended for AI visibility)

If you want ChatGPT to cite your content:

User-agent: GPTBot
Allow: /

How to test if GPTBot is blocked

UltraScout's free AI SEO Score instantly checks your robots.txt and shows:

• Whether GPTBot is explicitly allowed, blocked, or missing
• Your current AI SEO Score impact
• Recommendations for fixing blocks

Important notes

• Order matters: Place GPTBot rules before any User-agent: * catch-all rules
• Caching: Robots.txt changes may take up to 24 hours for crawlers to respect
• OpenAI documentation: GPTBot respects noindex but not noai meta tags — use robots.txt for reliable blocking

What to do if you accidentally blocked GPTBot

Run the free AI SEO Score tool. If it shows GPTBot blocked, upgrade to the AI Search Readiness Audit (£29) for the exact robots.txt fix code.