Auditaar
Start free
← All posts
Data study

How many websites block AI crawlers in robots.txt, and which crawlers do they block? (data from 1,000 websites, November 2026)

Auditaar measured robots.txt, llms.txt and related signals across 1,000 websites to see how many sites block AI crawlers by name, which operators get blocked most, and how blocking changes with site rank.

By Auditaar · November 9, 2026 · 3 min read
AI Visibility · robots.txt · Data study
The short answer

Across 1,000 websites sampled from the Tranco top one million list, 86.7% publish a robots.txt file but only 22.1% name any AI crawler in it. Just 16.5% block at least one AI training crawler, and 6.6% block all seven that Auditaar checked. CCBot and Bytespider are blocked most often; Googlebot and bingbot are almost never touched.

Key takeaways
  • 86.7% of websites publish a robots.txt file, but only 22.1% name any AI crawler in it by name.
  • Just 16.5% of sites block at least one AI training crawler, and only 6.6% block all seven that Auditaar checked.
  • CCBot (Common Crawl) and Bytespider (ByteDance) are the most-blocked crawlers in the study, at 12.6% and 12.4% of sites.
  • OpenAI's GPTBot is blocked by 11.5% of sites, far more than the 2.8% that block its search counterpart, OAI-SearchBot.
  • 18.3% of sites publish an llms.txt file, well ahead of the 5.3% that publish the newer Content Signals opt-out in robots.txt.
  • Blocking an AI training crawler is most common among top-1,000 sites (20.8%) and least common in the 10,001-to-100,000 band (11.2%).
Contents
  1. How many websites block AI crawlers in robots.txt?
  2. Which individual AI crawlers get blocked the most?
  3. Does AI-crawler blocking change with site rank?
  4. What else are sites publishing for AI crawlers?
  5. How this study was measured
  6. Sources

How many websites block AI crawlers in robots.txt?

Auditaar read robots.txt on 1,000 websites stratified across four rank bands of the Tranco top one million list. Most publish the file; far fewer name an AI crawler in it directly.

Fig. 1Robots.txt and AI crawlers across 1,000 websites
  • Publish a robots.txt file86.7%
  • Name at least one AI crawler22.1%
  • Block at least one AI training crawler16.5%
  • Block all 7 AI training crawlers checked6.6%
  • Block at least one AI assistant crawler6.3%

Source: Auditaar data study, November 2026, 1,000 websites

Which individual AI crawlers get blocked the most?

Named-crawler blocking varies widely by operator. Common Crawl's and ByteDance's bots draw the most blocks; Google's and Microsoft's own search crawlers draw almost none.

Table 1Share of sites blocking each named AI crawler
CrawlerOperatorBlocked (of all sites)
CCBotCommon Crawl12.6%
BytespiderByteDance12.4%
GPTBotOpenAI11.5%
ClaudeBotAnthropic10.7%
Google-ExtendedGoogle9.7%
Meta-ExternalAgentMeta9.7%
Applebot-ExtendedApple9%
AmazonbotAmazon8.3%
PerplexityBotPerplexity5%
ChatGPT-UserOpenAI4.9%
OAI-SearchBotOpenAI2.8%
Claude-SearchBotAnthropic2.8%
Claude-UserAnthropic2.8%
Perplexity-UserPerplexity2.6%
ApplebotApple1.6%
bingbotMicrosoft0.9%
GooglebotGoogle0.5%

Source: Auditaar data study, November 2026, 1,000 websites

Does AI-crawler blocking change with site rank?

Blocking peaks among the very largest sites, falls through the middle of the rank order, and ticks back up again outside the top 100,000.

Fig. 2Sites blocking at least one AI training crawler, by rank band
  • Top 1,00020.8%
  • 1,001 to 10,00018.8%
  • 10,001 to 100,00011.2%
  • 100,001 to 1,000,00015.2%

Source: Auditaar data study, November 2026, 1,000 websites

What else are sites publishing for AI crawlers?

Robots.txt is not the only file AI tools look for. Two newer ones are catching on at very different rates.

Fig. 3llms.txt: a plain-text map of a site's priority pages for AI toolsA minimal llms.txt points AI tools at the pages worth reading first.
llms.txt
# Northwind Analytics

> Northwind is a product analytics platform for
> subscription businesses.

## Docs
- [Getting started](/docs/start): install and first event
- [Pricing](/pricing): plans and limits
- [API reference](/docs/api): endpoints and auth

## Guides
- [Reducing churn](/guides/churn): the playbook

A minimal llms.txt: a plain-text map of your most useful pages, in the format AI models are learning to read.

Table 2Beyond blocking: other AI-facing files and signals
SignalShare of sitesScope
Publish an llms.txt file18.3%of all 1,000 sites
Publish Content Signals in robots.txt5.3%of sites where robots.txt was checked
Declare a sitemap in robots.txt74%of sites with a robots.txt file

Source: Auditaar data study, November 2026, 1,000 websites

How this study was measured

Every number above comes from one Auditaar crawl of a stratified random sample of the Tranco top one million list, collected on November 5, 2026.

Table 3Sample and collection, by rank band
Rank bandWebsites drawnWebsites measured
Top 1,000504250
1,001 to 10,000423250
10,001 to 100,000390250
100,001 to 1,000,000377250

Source: Auditaar data study, November 2026, Tranco list JZPGY

  • 268 replacement draws found no website at all at the domain.
  • 196 connection attempts timed out or failed outright.
  • 173 sites refused the request outright, such as a block, a challenge, or an error that marked the site unreachable.
  • 44 sites were excluded because their own robots.txt opted the audit crawler out directly.
  • 7 sites had only a private network address, and 6 did not return an HTML page.
  • One homepage per site was read, following redirects, so these figures describe home pages, not whole sites.
  • robots.txt was honored during collection: a site that disallowed the audit crawler itself, or all crawlers, was excluded and replaced, which can undercount sites that already block broadly as a policy.
  • Only sites that answered over HTTPS with an HTML page were measured, so HTTP-only sites are not represented.
  • Each rank band carries equal weight in the all-band figures above, so they describe this sample evenly across bands, not the web weighted by traffic.

Run a free AI visibility check to see whether your own robots.txt blocks an AI crawler you didn't mean to, or leaves one open you meant to close.

Sources

  1. Tranco list JZPGY

See it on your own site

Auditaar turns a single URL into a scored, sourced, ordered plan across all six pillars, AI Visibility included.

Audit my websiteSee the product

Free AI visibility check. No sign-up required.

Related reading

AI Visibilityrobots.txt

Should you block AI crawlers like GPTBot, ClaudeBot or PerplexityBot?

Blocking an AI crawler in robots.txt does something different depending on the kind of bot it is: here is what each of the 17 named AI agents does, and how to write a robots.txt that keeps a site citable without opting into training it never agreed to.

September 28, 2026 · 4 min read
AI VisibilityGEO

Auditaar vs the AI-visibility trackers

Dedicated AI-visibility trackers tell you whether AI assistants cite you. Auditaar does that and audits the rest of your site and hands back the fixes, from a URL, with a free check. Here is how they compare.

November 12, 2026 · 3 min read
AI VisibilityGEO

The best AI visibility tools in 2026

AI visibility tools track whether AI assistants like ChatGPT, Google's AI Overviews and Perplexity can find and cite your site. Here are the main options in 2026, from free checks to enterprise platforms, and how to choose.

November 11, 2026 · 4 min read
AI VisibilityAEO

What is an AI-Visibility Score, and how is it measured?

An AI-Visibility Score measures whether answer engines like ChatGPT and Google's AI Overviews actually cite and name your site. Here is what goes into one, and how Auditaar reads it.

August 25, 2026 · 4 min read