Your robots.txt Is Accidentally Blocking the AI Traffic You Want

The one-time decision nobody revisited

When AI crawlers first showed up in logs, the reflex for many teams was to block them, protect the content, deny the scrapers, do not feed the machine for free. It felt prudent. The problem is that the calculus inverted: those same crawlers are now the discovery and citation pipeline for a fast-growing share of high-intent buyers. The "block them" line you added to defend your content is now a "block them" line that erases you from AI answers.

And almost nobody revisits robots.txt. It is a file you write once and forget, often inheriting rules from a template, a framework default, a security audit, or a copied snippet. The result is a quiet, unexamined policy on the single most important question in AEO: can the answer engines read me at all? Per the Cloudflare-network robots.txt report, among thousands of files parsed, GPTBot was mentioned in ~13.8% of robots.txt files and ClaudeBot in ~11.5%, and a meaningful slice of those mentions are outright blocks (GPTBot fully disallowed by ~5.2% of sites, Google-Extended ~4.2%).

Know your bots: who is who

Not all AI user agents do the same thing, and conflating them leads to bad policy. There is a critical distinction between crawlers that gather training data and crawlers that fetch pages in real time to answer a user's question right now. You might rationally block the former and absolutely want the latter.

• GPTBot, OpenAI's crawler for gathering training data. Blocking it affects future model training, not necessarily live ChatGPT browsing.
• OAI-SearchBot / ChatGPT-User, OpenAI's agents for surfacing and fetching pages to answer live queries. Blocking these removes you from ChatGPT's real-time answers, usually the opposite of what you want.
• ClaudeBot, Anthropic's crawler. Soar's AI bots guide notes ClaudeBot blocking grew fastest among major bots through early 2026.
• PerplexityBot, Perplexity's crawler for its answer engine, which cites sources prominently.
• Google-Extended, a control token that governs use of your content for Gemini/Vertex AI training. Crucially, blocking it does not affect normal Google Search indexing.
• CCBot, Common Crawl, a public dataset many models train on.

The nuance that trips teams up: blocking Google-Extended is sometimes confused with blocking Google Search. They are separate. Likewise, blocking GPTBot (training) is different from blocking OAI-SearchBot (live answers). A blanket "block all AI" rule frequently blocks the live-answer agents you most want, while a half-measure leaves training crawlers you meant to block.

The cost of getting it wrong

This is not a symmetric "you might lose a little" trade. The traffic data is stark. A study by researchers at Rutgers Business School and The Wharton School, summarized in the same robots.txt report, found that publishers blocking AI crawlers via robots.txt experienced a 23.1% decline in total monthly visits and a 13.9% decline in human-only browsing. The blocked crawler does not just cost you AI referrals, it correlates with a broad traffic decline as your presence in AI-mediated discovery shrinks.

The flip side, accidental over-exposure, has its own cost, mostly server load and unwanted training use. But for the large majority of businesses whose goal is to be found and recommended, the expensive mistake is blocking, not allowing. The point is to look, decide, and configure intentionally rather than inherit a default that silently caps your AEO upside.

Read your own robots.txt the way the bots do

The first step is mechanical: fetch your robots.txt and resolve, for each AI user agent, whether your key URLs are allowed. The rules are evaluated per-user-agent with longest-match precedence, so eyeballing the file is error-prone. Check it programmatically.

# Resolve allow/deny per AI user agent against your robots.txt
import urllib.robotparser as urp

AI_BOTS = ['GPTBot', 'OAI-SearchBot', 'ChatGPT-User', 'ClaudeBot',
           'PerplexityBot', 'Google-Extended', 'CCBot', 'Googlebot']

rp = urp.RobotFileParser()
rp.set_url('https://example.com/robots.txt')
rp.read()

test_paths = ['/', '/products/', '/blog/flagship-post', '/pricing']
for bot in AI_BOTS:
    verdict = {p: rp.can_fetch(bot, p) for p in test_paths}
    blocked = [p for p, ok in verdict.items() if not ok]
    status = 'BLOCKED on ' + ', '.join(blocked) if blocked else 'allowed'
    print(f'{bot:18} {status}')
# Surprises here, like OAI-SearchBot BLOCKED while you want
# ChatGPT to recommend you, are exactly the point of the scan.

A deliberate robots.txt for an AEO-positive site

For most businesses that want to be discovered and cited, the right posture is: allow the live-answer and citing crawlers, decide consciously about pure-training crawlers, and never accidentally block Search. Here is a worked example with the reasoning inline.

# robots.txt, deliberate AI crawler policy
# Goal: be recommended and cited by answer engines.

# Live-answer + citing agents: ALLOW (you want these)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /

# Training crawlers: your call. Allowing builds future presence;
# blocking protects content from training. Decide on purpose.
User-agent: GPTBot
Allow: /
User-agent: CCBot
Allow: /
User-agent: Google-Extended
Allow: /

# Never block normal Search by mistake.
User-agent: Googlebot
Allow: /

# Generic fallback
User-agent: *
Allow: /
Disallow: /cart/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

The bottom line

Your robots.txt is making an AEO decision whether or not you meant to make one. With AI crawlers now a primary discovery and citation channel, an inherited "block the scrapers" rule can erase you from ChatGPT and Perplexity answers, and the data ties AI-crawler blocking to a ~23% drop in total traffic. The fix is not "allow everything" or "block everything, " it is decide on purpose: distinguish training crawlers from live-answer agents, never accidentally block Search, resolve your current policy programmatically per user agent, and re-scan on a schedule as new bots appear. The expensive mistake is the one you never looked at.