Block AI Crawlers or Allow Them: 2026 Site Owner Guide

The short answer

You control AI crawlers in two layers. A robots.txt file tells well-behaved bots like GPTBot and ClaudeBot what they may access, and a network tool like Cloudflare enforces that decision against bots that never ask. Since Cloudflare's July 1, 2026 announcement, the enforcement layer has real teeth: AI training and agent crawlers will be blocked by default on new domains starting September 15, and every Cloudflare customer, including the Free plan, can now allow or block AI crawlers by category from the dashboard.

I run clskillshub.com and this post is the exact setup I use, plus the copy-paste robots.txt block most people came here for.

What did Cloudflare change about AI crawlers in July 2026?

On July 1, 2026, Cloudflare replaced its single block-AI-bots switch with three categories of AI traffic, laid out in its official announcement:

1. Search: bots that collect or index your content so an AI product can answer questions about it later, usually with a link back. OAI-SearchBot and Claude-SearchBot live here.
2. Agent: automated systems acting in real time on a person's behalf, like ChatGPT fetching your pricing page because a user asked about it.
3. Training: crawlers pulling your content into a model's weights. GPTBot, ClaudeBot, and CCBot are the big names.

Two details matter more than the categories themselves. First, on September 15, 2026, new domains onboarding to Cloudflare get Training and Agent blocked by default on pages that display ads, while Search stays allowed. That is the permission flip the trade press has been covering: for a meaningful slice of the web, AI crawlers now need permission instead of enjoying it by default. Second, multi-purpose crawlers are evaluated under all of their behaviors. Googlebot crawls for both search and Google's AI features, so a blanket Training block through these controls can catch Googlebot and hurt your regular rankings. That one footnote should shape your entire decision.

The category controls are live now on every plan, Free included. Existing domains keep their current settings; only newly onboarded domains get the September 15 defaults automatically. Cloudflare also runs pay per crawl, which lets you charge AI crawlers per request instead of blocking them outright. It is still in limited rollout, so treat it as something to watch rather than something to configure today.

Should you block AI crawlers or allow them?

The honest answer is that blanket blocking and blanket allowing are both usually wrong, because the three bot types give you completely different deals.

Training crawlers give you almost nothing back. Your content improves someone else's model, uncredited, and no visitor ever arrives because of it. If your content is your product, blocking training crawlers is easy to justify.

AI search crawlers are a different trade. If OAI-SearchBot or PerplexityBot can index you, your pages can get cited inside ChatGPT and Perplexity answers, with links. On clskillshub.com that referral traffic is a trickle compared to Google, but the visitors arrive with intent, because someone who clicks through from an AI answer already has the exact problem the page solves. Blocking AI search makes you invisible in the fastest-growing answer surfaces on the web. We invest in ranking there deliberately; the workflow is in my post on using Claude for SEO content.

User-initiated agent fetches sit in the middle. A human asked for your page right now. Blocking those rarely protects anything and mostly just breaks the experience for a real person who wanted your content.

So the three defensible positions are: block everything (your content is the paid product and scraping is pure loss), allow everything (distribution matters more than protection), or split the difference (block training, allow search and agents). Most small sites should split. More on that at the end.

How do you block AI crawlers with robots.txt?

Here is the complete block. Every user-agent string below is verified against the official documentation from OpenAI, Anthropic, and Perplexity, plus Google's, Apple's, Meta's, and Common Crawl's crawler docs, as of July 2026. Paste it into the robots.txt at your domain root and delete the sections that do not match your policy.

# --- AI training crawlers (content used to train models) ---
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

# --- AI search indexing (keeps you citable in AI answers) ---
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# --- User-initiated fetches (a human asked for your page) ---
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

# --- Everyone else, including regular search engines ---
User-agent: *
Allow: /

Three notes on the training section. Google-Extended is not a separate crawler; it is a control token that Googlebot reads, so disallowing it stops Google from using your content for Gemini training without touching your search rankings. Applebot-Extended works the same way for Apple's foundation models. Bytespider is ByteDance's crawler, and ByteDance publishes no opt-out documentation; multiple infrastructure companies have reported it ignoring robots.txt, so the line above is a statement of policy, not a guarantee. Enforcement for bots like that comes at the network layer, covered next.

How do you block AI crawlers in Cloudflare?

If your site sits behind Cloudflare, the dashboard now does most of the work:

1. Log in, select your domain.
2. Open the zone settings and find the AI crawler controls with the three categories: Search, Agent, Training.
3. Set each category to allow or block. If you want the split policy, block Training and allow the other two.
4. Check the AI crawl analytics to see which bots are actually hitting you before and after. Cloudflare's AI Crawl Control view lists activity per crawler and lets you override individual bots.
5. If you rely on Google traffic, verify Googlebot is not caught by a Training block, since multi-purpose crawlers are judged on all their behaviors.

Enterprise Bot Management customers additionally get content use signals with three levels: immediate (no storage or reuse), reference (indexing, excerpts, and links back, the default), and full (summaries and reproduction allowed). These extend robots.txt through Content Signals rather than replacing it.

Keep the robots.txt from the previous section in place even with Cloudflare configured. The file is your declared policy that compliant bots read; Cloudflare is the wall for everyone else.

How do you control AI crawlers on WordPress, Vercel, and Netlify?

WordPress: a physical robots.txt in your web root overrides WordPress's virtual one. Either upload the file directly, or paste the block into the robots.txt editor in Yoast or All in One SEO. If the site also sits behind Cloudflare, do both.

Vercel: there is no AI bot toggle in the dashboard as of this writing. Commit a public/robots.txt, or generate one from app/robots.ts in Next.js. For enforcement, Vercel's firewall supports custom rules matched on user agent.

Netlify: drop robots.txt into your publish directory. For enforcement, an edge function can return 403 on matching user agents.

A VPS with nginx, which is what clskillshub.com runs after our Vercel to Hostinger migration: serve robots.txt from the root and add a user-agent rule for the bots that ignore the file. Mine is four lines in the server block:

if ($http_user_agent ~* "Bytespider") {
    return 403;
}

That is the whole pattern. Declare policy in robots.txt, enforce the exceptions in nginx.

What is llms.txt and does it matter?

llms.txt is a proposed convention: a markdown file at your domain root that lists your most important pages in a format meant for language models to read. It is a menu, not a lock. It does not allow or block anything, and it is not related to robots.txt or Cloudflare's controls.

As of July 2026, none of the major AI companies document honoring llms.txt in their crawler behavior. I keep one on clskillshub.com anyway because it costs ten minutes and might help an agent land on the right page instead of a random one. Just do not confuse publishing an llms.txt with having a crawler policy. It controls nothing.

Can AI companies ignore robots.txt?

Mechanically, yes. robots.txt is a request, not a lock, and nothing technical stops a crawler from reading your file and fetching your pages anyway.

Here is the documented reality. OpenAI, Anthropic, Google, Apple, Meta, Common Crawl, and Perplexity all state their training and search crawlers honor robots.txt. The gray zone is user-initiated fetching, and it is a gray zone by design: OpenAI says ChatGPT-User is not used for automatic crawling and that robots rules may not apply to it, and Perplexity's docs state plainly that Perplexity-User generally ignores robots.txt because a human requested the page. Anthropic says Claude-User does honor it. And then there is the long tail of crawlers, Bytespider most famously, that simply do not participate in the honor system at all.

This is exactly why Cloudflare's July change matters more than another robots.txt tutorial. It moves a large slice of the web from an honor system to a permission system, enforced at the network edge where a bot's compliance is not optional. If a crawler ignoring your robots.txt would genuinely hurt your business, the file alone was never enough. You need the wall too.

What should most small site owners actually do?

Block training, allow AI search and user-initiated fetches, and enforce with whatever network layer you already have. That is the whole recommendation. Training crawls give you nothing back; AI search sends humans; agent fetches are humans.

Two exceptions. If your content is the paid product, gated courses, original data, journalism behind a meter, block the Agent category too, because an AI reading your page to a user is functionally a paywall bypass. And if you use Cloudflare's category toggles and depend on Google traffic, confirm before September 15 that a Training block is not catching Googlebot on your zone.

On clskillshub.com I run exactly the split above: the robots.txt block from this post with the search and user sections set to allow, plus the nginx rule for Bytespider. Total setup time was about twenty minutes, and most of that was reading the bot docs so you do not have to.

If you want the exact prompts I use to research and draft pages like this one, the free 75-page Claude guide is at clskillshub.com/guide.