Skip to content
AIReadyNow

Free tool

AI crawler robots.txt generator

Let the crawlers that put you in front of customers in, and make a deliberate choice about the ones that train models.

1. Search and AI answer engines

These build the indexes behind ChatGPT Search, Perplexity, Claude, Google and Bing results. For a business that wants customers, allow them.

2. Model training (your choice)

Blocking these doesn’t remove you from search or AI answers. It only opts your content out of future training data. Being in training data can help models know your brand, so for most local businesses leaving them unblocked is fine.

3. Your own rules

Your robots.txt

robots.txt
# Default rules for all crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Search and AI answer engines: allowed
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /

# Assistants fetching a page for a user: allowed
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-User
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /

How to install it

  1. Upload it to the root of your site so it loads at yoursite.com/robots.txt. WordPress: most SEO plugins have a robots.txt editor. Wix, Squarespace and Shopify manage it for you, so check their settings instead.
  2. Back up your current file first, and keep any rules you still need.
  3. Check your CDN or firewall too (Cloudflare’s “Bot Fight Mode” and “Block AI bots” settings can block crawlers even when robots.txt allows them).
  4. Re-run the free scan to confirm.

Note that each crawler follows only the most specific group that names it, which is why the allow groups above repeat your disallow rules.