How to Configure robots.txt for AI Search Engines & LLM Citations
Allow ChatGPT, Claude, and Perplexity to index and cite your content while controlling scraper access.
A GPTBot Disallowed notice means your robots.txt file blocks OpenAI's crawler ('User-agent: GPTBot') or SearchBot. When disallowed, AI search engines (SearchGPT, Perplexity, Claude) cannot read or cite your pages in AI Overviews and chat answers.
Root Cause Analysis
Websites often copy generic robots.txt templates that block all AI bots, unintentionally preventing search-oriented AI answer engines from citing their brand.
Step-by-Step Resolution Procedure
Run the AI Bot & llms.txt Scanner above to check permissions for GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot.
Allow search and citation bots (OAI-SearchBot, PerplexityBot) while optionally disallowing training scrapers.
Create an /llms.txt file at your site root linking key pages and tools with clean summaries for LLMs.
Verification Command or Config Pattern
User-agent: *
Allow: /
# Allow AI Search & Citation Bots
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xmlFrequently Asked Remediation Questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is used for AI model training and web browsing. OAI-SearchBot is specifically used by SearchGPT to discover and cite links in real-time search queries.
Does blocking GPTBot hurt my standard Google rankings?
No. Standard Googlebot uses its own user-agent. However, blocking Google-Extended prevents your content from appearing in Google Gemini training.
What is an /llms.txt file?
/llms.txt is an open proposal providing a concise, clean Markdown directory of a website's purpose, key links, and documentation specifically formatted for LLMs.
How do I check if my site is AI-ready?
Use the AI Bot & llms.txt Scanner on OnChainScans to evaluate crawler policies and llms.txt availability in real time.
Complementary Diagnostic Scanners
All 34 toolsWCAG Accessibility & Contrast Scanner
Audits foreground/background luminosity contrast ratios against WCAG 2.2 AA (4.5:1) and AAA (7:1) thresholds, detecting missing alt attributes and keyboard traps.
Robots.txt & Sitemap.xml Scanner
Verifies syntax of robots.txt files, parses nested sitemap_index.xml structures, and audits for broken URLs, conflicting directives, and 404 indexing errors.
Website Font, Color & Asset Scanner
Crawls live stylesheets to extract distinct hex color palettes, webfont load methods (WOFF2/FOUT metrics), and total uncompressed asset payload distribution.
Related Troubleshooting & RFC Playbooks
How to Fix SPF PermError: 'Too Many DNS Lookups' (RFC 7208)
Resolve the 10-lookup limit that causes receiving mail servers (Gmail, Outlook) to reject your emails.
How to Fix NET::ERR_CERT_AUTHORITY_INVALID SSL Handshake Errors
Repair broken intermediate certificate bundles and establish a trusted X.509 chain of trust.
How to Fix ERR_TOO_MANY_REDIRECTS & Circular Redirect Loops
Diagnose infinite HTTP 301/302 redirects between Cloudflare Flexible SSL, Nginx, and CMS canonical rules.