Robots.txt declares which crawlers may fetch your pages. Reputable AI crawlers honor it, but the file itself has no technical power to enforce anything. If you want real control, you need per-user-agent rules in robots.txt plus server-level backup like a WAF or authentication for anything truly sensitive.
TL;DR:
- Robots.txt is a declaration that well-behaved AI crawlers usually respect but has no power to prevent uncooperative scrapers from ignoring its rules.
- Different AI bots serve various purposes, so site owners should set specific rules to allow or block individual user-agents based on their business goals.
- Using authentication and other server-level defenses is essential for truly sensitive content, as robots.txt alone cannot prevent unauthorized scraping.
- Regularly testing, monitoring, and verifying IP ranges help ensure that your robots.txt rules are effective and that unwanted crawling or scraping is minimized.
- Balancing content visibility and protection requires tailored rules aligned with your objectives, along with ongoing review and technical safeguards.
Table of Contents
- What Robots.txt Actually Does for AI Crawlers
- AI Crawler User-Agents You Need to Know
- How to Write Robots.txt Rules for AI Bots
- Why Robots.txt Alone Won’t Stop a Determined Scraper
- Testing and Monitoring: Make Sure Your Rules Actually Work
- Which Robots.txt Setup Fits Your Business Goals
- How Stellor Tracks AI Crawler Behavior Every Week
- A Practical Take on Balancing Visibility and Protection
- Sources
- FAQ
What Robots.txt Actually Does for AI Crawlers
Robots.txt is a plain text file at your domain root (yoursite.com/robots.txt) that tells crawlers which paths they may or may not fetch. The syntax is simple: User-agent names the bot, Disallow and Allow set paths, and Sitemap points to your XML sitemap. That’s the entire toolkit, standardized in RFC 9309.
Here’s the part people miss: robots.txt is a declaration, not a lock. It tells well-behaved crawlers where not to go. It does nothing to stop a script that ignores the rules. Google Search Central is explicit about this: a disallowed URL can still turn up in search results if other sites link to it. If you actually need a page hidden, use noindex or password protection, not a Disallow line.
AI Crawler User-Agents You Need to Know
Not every AI bot wants the same thing from your site, and treating them as one blob is the most common robots.txt mistake site owners make. Some are indexing your content for a live answer engine. Some are training a model that will never send you a click. Others fetch a single page in real time because a user pasted your URL into a chat window.
The tokens worth knowing:
- GPTBot — crawls content for OpenAI’s model training.
- OAI-SearchBot — powers search-style results inside ChatGPT; distinct from GPTBot, and OpenAI treats each token independently.
- ChatGPT-User — fetches a page live when a user asks ChatGPT about a specific URL.
- Google-Extended — controls whether Google can use your content to train Gemini and AI Overviews, separate from regular Googlebot indexing.
- ClaudeBot — Anthropic’s crawler, used for both training and retrieval depending on context.
- PerplexityBot — indexes pages for Perplexity’s answer engine.
- Applebot-Extended — governs Apple’s generative AI training use of your content.
The practical rule: allow bots that can send you referral traffic or citations, and think hard before blocking anything that drives visibility in AI search results. Block bots that only harvest content for training with no return traffic, if that tradeoff matters to your business.
How to Write Robots.txt Rules for AI Bots
Allowing one OpenAI token does not allow another. Each user-agent needs its own explicit block, and OpenAI’s own documentation confirms allow/disallow decisions apply per token, not globally. Here are three templates you can adapt right now.
Template A: Balanced (recommended for most sites) — Keep search and live-answer bots open for citation potential, block pure training crawlers.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
Sitemap: https://yoursite.com/sitemap.xml
Template B: Strict opt-out — For proprietary content, research, or paywalled material where you want no AI ingestion at all. Block every known token, training and live-fetch alike.
Template C: Fully permissive — For businesses chasing maximum AI visibility, where being cited matters more than protecting the raw content. Allow all major AI user-agents with a blanket Allow: /.
A few mechanical notes: more specific rules beat general ones, order within a User-agent block doesn’t override specificity, and if you already have a robots.txt file, append new blocks rather than replacing the whole thing. If you run a CDN like Cloudflare, check whether its managed robots.txt feature is active. It can merge with or prepend to your existing file automatically, which sometimes surprises site owners who assumed their manual edits were the only source of truth.
Why Robots.txt Alone Won’t Stop a Determined Scraper
Robots.txt runs on trust. A legitimate crawler reads the file and complies. A scraper that doesn’t care simply ignores it, and there’s no technical mechanism in the protocol itself to stop that. This is why Cloudflare’s guidance pairs its managed robots.txt with actual enforcement: WAF rules, IP deny-lists, and rate limiting for anything that needs to be genuinely locked down.
If a page holds sensitive pricing data, unpublished research, or gated client content, robots.txt is the wrong tool entirely. Put it behind authentication. For everything else, layer defenses: rate-limit aggressive scrapers, restrict by referrer where it makes sense, and reserve login walls for content you truly cannot afford to have scraped. Also expect delay. CDN and edge caching mean a robots.txt update doesn’t always take effect instantly across every server pulling your file.
Testing and Monitoring: Make Sure Your Rules Actually Work
Writing the rules is the easy part. Confirming crawlers respect them takes ongoing checking.
- Verify with IP ranges, not just user-agent strings. User-agent headers are trivial to spoof. OpenAI publishes IP ranges for GPTBot and its other crawlers; cross-check log entries claiming to be GPTBot against those ranges before trusting them.
- Grep your server logs for known tokens. Pull every request with GPTBot, ClaudeBot, PerplexityBot, and similar strings, then correlate against vendor IP lists to separate real crawlers from impersonators.
- Give changes time to propagate. OpenAI notes it can take about 24 hours for its systems to pick up a robots.txt update, so don’t panic if a block doesn’t show results the same day.
- Run a third-party AI crawler access checker after any change to confirm the rule is live and parsed correctly, not just saved.
- Set alerts for unfamiliar user-agents or traffic spikes. A sudden surge from an unrecognized bot is often the first sign of a scraper ignoring your file entirely.
Pro Tip: Schedule a recurring 15-minute robots.txt and log review every month. Crawler tokens change, new AI vendors launch, and a rule that made sense in January can quietly go stale by summer.
Which Robots.txt Setup Fits Your Business Goals
Your ideal configuration depends on one question: is your priority visibility or protection?
- Maximize AI citations: Use Template C or Template A. Keep OAI-SearchBot, ChatGPT-User, and PerplexityBot open so your business shows up when someone asks an AI engine for a recommendation.
- Protect proprietary content: Use Template B. Block every known AI token, and back it with authentication for anything genuinely confidential.
- Balanced approach (most local and service businesses): Template A. Allow the referral-driving bots, disallow the pure training crawlers, and revisit quarterly.
Operational shortcuts help here. Cloudflare’s managed robots.txt keeps its AI bot list current automatically, and the community-maintained ai.robots.txt list is a solid starting point for a strict block list, though vendor docs and IP verification should always have the final say.
| Goal | Recommended template | Key bots to allow | Key bots to block |
|---|---|---|---|
| Maximize AI citations | Template C | OAI-SearchBot, ChatGPT-User, PerplexityBot | None |
| Protect proprietary content | Template B | None | GPTBot, ClaudeBot, Google-Extended, PerplexityBot |
| Balanced (most businesses) | Template A | OAI-SearchBot, ChatGPT-User, PerplexityBot | GPTBot, Google-Extended, ClaudeBot |
How Stellor Tracks AI Crawler Behavior Every Week
At Stellor, weekly technical audits check robots.txt syntax alongside schema markup and page speed, because a broken rule can quietly cut off an entire AI citation channel. The platform also runs LLM visibility tracking across ChatGPT, Claude, Perplexity, and Gemini, so you can see whether your content is actually getting cited, not just crawled.
Key metrics to watch include AI citation count by engine, crawler hits broken down by user-agent, and blocked-training attempts that show up in logs. A simple checklist covers most of it: confirm robots.txt loads without errors, check for new unrecognized user-agents, and re-verify that referral-driving bots remain open after any site migration or local SEO update.
A Practical Take on Balancing Visibility and Protection
Most site owners overcorrect in one direction. They either block everything AI-related out of fear or leave the file wide open and never look at it again. Neither serves you. Keep the bots that can actually send you traffic or citations unblocked, lock down the handful of pages that genuinely need protecting with real technical controls, and check your logs monthly. Robots.txt is a policy statement, not a security system, and treating it as either extreme wastes the leverage it does have.
— Cole
Sources
- Robots.txt Introduction and Guide | Google Search Central
- Overview of OpenAI Crawlers
- robots.txt setting · Cloudflare bot solutions docs
- RFC 9309 (Robots Exclusion Protocol)
FAQ
Does Robots.txt Stop AI Crawlers?
Robots.txt only stops crawlers that choose to honor it. Reputable AI crawlers like GPTBot and ClaudeBot generally comply with published rules, but robots.txt has no technical mechanism to block a scraper that ignores the file, according to Google Search Central.
Is Robots.txt Still Used?
Yes, robots.txt remains the standard first line of defense for managing crawler access, and it’s still codified in the current RFC 9309 spec. Every major search engine and AI vendor, including OpenAI and Google, checks it before crawling.
How Do I Fix a “Blocked by Robots.txt” Error?
Check your robots.txt file for a Disallow rule matching the affected URL path, and remove or narrow it if the page should be crawlable. Then resubmit the URL in your search console tool and allow time for re-crawling, since some systems take up to 24 hours to reflect changes.
Is Ignoring Robots.txt Illegal?
Ignoring robots.txt isn’t illegal on its own; it’s a voluntary protocol, not a legal mandate, as Cloudflare’s documentation notes. That said, scraping content against a site’s stated preferences can still raise separate legal issues around copyright or unauthorized access depending on how the content is used.
Should I Block Google-Extended if I Still Want Google Search Traffic?
Yes, you can block Google-Extended without affecting regular Googlebot indexing. Google-Extended only controls whether your content trains Gemini and AI Overviews, and it’s separate from the crawler that builds your standard search listings.