Guides · 6 min read
Are you blocking AI crawlers? How to check and fix it
One of the most common reasons a business never shows up in AI answers is also the easiest to fix: the crawlers that feed those answers are being turned away. Sometimes that’s deliberate. More often it’s a leftover rule, a security plugin, or a CDN setting nobody remembers enabling.
Three kinds of AI crawler
- Search crawlers build the indexes that answers are drawn from. Blocking these makes you harder to find.
- User-triggered agents fetch a page live when a person asks an assistant about it or pastes your link.
- Training crawlers collect content that may be used to train future models. Blocking these is a privacy and licensing choice, and doesn’t remove you from search results.
| Crawler | Company | What it does |
|---|---|---|
OAI-SearchBot | OpenAI | Builds the index used for ChatGPT search results |
ChatGPT-User | OpenAI | Opens a page when a ChatGPT user asks about it |
GPTBot | OpenAI | Collects content that may be used to train models |
PerplexityBot | Perplexity | Builds Perplexity’s search index |
Claude-SearchBot | Anthropic | Improves search result quality in Claude |
Claude-User | Anthropic | Opens a page when a Claude user asks about it |
ClaudeBot | Anthropic | Collects content that may be used to train models |
Googlebot | Google Search, which also feeds Google’s AI Overviews and AI Mode | |
Google-Extended | A control token for using content in Gemini; does not affect Search | |
Bingbot | Microsoft | Bing, which feeds Copilot and some other AI products |
How to check
- Open
yoursite.com/robots.txtin a browser. Look forDisallow: /underUser-agent: *or under any of the names above. - Run the free scan. It reads your robots.txt and reports which crawlers are allowed.
- Check your security layer: Cloudflare (including “Bot Fight Mode” and any AI-bot blocking toggles), Wordfence, Sucuri, and your host’s own bot protection can block crawlers regardless of what robots.txt says.
A sensible default for a local business
Allow search crawlers and user-triggered agents. Decide on training crawlers deliberately. Many businesses leave them allowed so models learn their name; others block them. Either is reasonable, as long as it’s a choice. Use the robots.txt generator to produce a clean file.
Don’t forget JavaScript
Many AI crawlers fetch raw HTML and don’t run JavaScript. If your page is empty until scripts execute, the crawler sees an empty page. Make sure your key text (what you do, where you are, how to contact you) is in the HTML itself. This very site is pre-rendered for that reason.