OnChainScans
GPTBot / AI Crawler Disallowed in robots.txt•The Robots Exclusion Protocol (RFC 9309) & /llms.txt Standard• LIVE SCANNER EMBEDDED

How to Configure robots.txt for AI Search Engines & LLM Citations

Allow ChatGPT, Claude, and Perplexity to index and cite your content while controlling scraper access.

Live Diagnostic Tool: AI Bot & llms.txt Access ScannerFull Tool Page
Executive Answer & Impact Analysis

A GPTBot Disallowed notice means your robots.txt file blocks OpenAI's crawler ('User-agent: GPTBot') or SearchBot. When disallowed, AI search engines (SearchGPT, Perplexity, Claude) cannot read or cite your pages in AI Overviews and chat answers.

Root Cause Analysis

Websites often copy generic robots.txt templates that block all AI bots, unintentionally preventing search-oriented AI answer engines from citing their brand.

Step-by-Step Resolution Procedure

1Audit AI Crawler Access Matrix

Run the AI Bot & llms.txt Scanner above to check permissions for GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot.

2Distinguish Training Scrapers from Search Bots

Allow search and citation bots (OAI-SearchBot, PerplexityBot) while optionally disallowing training scrapers.

3Publish an /llms.txt Index

Create an /llms.txt file at your site root linking key pages and tools with clean summaries for LLMs.

Verification Command or Config Pattern

TERMINAL / CONFIGRFC The Robots Exclusion Protocol (RFC 9309) & /llms.txt Standard
User-agent: *
Allow: /

# Allow AI Search & Citation Bots
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

Frequently Asked Remediation Questions

What is the difference between GPTBot and OAI-SearchBot?

GPTBot is used for AI model training and web browsing. OAI-SearchBot is specifically used by SearchGPT to discover and cite links in real-time search queries.

Does blocking GPTBot hurt my standard Google rankings?

No. Standard Googlebot uses its own user-agent. However, blocking Google-Extended prevents your content from appearing in Google Gemini training.

What is an /llms.txt file?

/llms.txt is an open proposal providing a concise, clean Markdown directory of a website's purpose, key links, and documentation specifically formatted for LLMs.

How do I check if my site is AI-ready?

Use the AI Bot & llms.txt Scanner on OnChainScans to evaluate crawler policies and llms.txt availability in real time.