For most small businesses, the answer is yes — allow the crawlers that fetch your pages to answer live questions, and decide separately about the ones that harvest text for training. They are not the same bots, and collapsing them into one decision is where people go wrong.
The instinct to block is understandable. You wrote the words. A machine is reading them and giving away the substance without sending anyone to you. Blocking feels like the obvious defensive move.
But it's worth being precise about what you'd actually be defending, because the two kinds of crawler have almost opposite consequences.
Training crawlers versus retrieval crawlers
A training crawler collects text that may be used to train a future model. Whatever it takes is absorbed permanently. There's no link, no attribution, no way to remove it later, and no visitor. If your objection is "I don't want my writing inside someone's model," this is the one you object to.
A retrieval crawler fetches your page at the moment a user asks a question, so the assistant can quote something current and name the source. This is the one that produces the citation with your name on it. This is how you appear in an AI answer at all.
Roughly how they divide:
GPTBot— OpenAI, trainingOAI-SearchBot— OpenAI, retrieval for ChatGPT searchClaudeBot— Anthropic, trainingClaude-SearchBot— Anthropic, retrievalPerplexityBot— Perplexity, primarily retrievalGoogle-Extended— Google, controls Gemini and AI training onlyApplebot-Extended— Apple, trainingCCBot— Common Crawl, an open dataset many models draw from
Google-Extended does not affect your Google Search rankings — it only controls Gemini and AI training. But blocking Googlebot removes you from Google Search entirely. They are different user agents and people confuse them constantly. Check which one is in your file.
The case for allowing retrieval
The honest framing is that this is the same bargain you already accepted with search engines, and you accepted it because it works.
Google has always crawled your content for free, shown parts of it in the results, and answered a good share of queries without sending anyone through. Featured snippets have done exactly what people fear from AI for over a decade. Nobody blocks Googlebot over it, because the trade — some of your content, in exchange for being findable at all — is obviously worth taking.
Retrieval crawlers are the same trade in a new interface. The customer asks their question in a chat window instead of a search box. If you're blocked, you're not in the answer, and the competitor who allowed it is.
There's also a quieter cost. When a model has no current information about you, it doesn't say "I don't know." It falls back on whatever it absorbed in training, which may be years stale, or it infers something plausible and wrong. I've watched an assistant confidently give a business the wrong service list. Being absent doesn't protect your reputation — it just removes your ability to correct the record.
The case for blocking training
This one is more legitimate and depends entirely on what your content is.
If your writing is the product — a paid course, an original research corpus, a body of work whose value is the words themselves — then training absorption is a genuine loss with nothing offered back. Block it.
If your content is marketing for a service you sell, the calculus flips. Your articles exist to demonstrate competence so someone hires you. A model that has absorbed your explanation of technical SEO and repeats your framing is doing marketing for you, badly attributed but at scale. The words were never the asset. The work is.
Most small businesses are firmly in the second category, which is why most should allow both and stop worrying about it.
What to actually put in robots.txt
The default I'd recommend for a service business — allow everything, and say so explicitly rather than relying on silence:
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://example.com/sitemap.xmlIf you want to keep your writing out of training data while staying citable, block the training bots only:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Two caveats you should know before trusting either file. robots.txt is a request, not a fence — well-behaved crawlers honour it and badly-behaved ones ignore it, and there is no enforcement. And this list changes: bots get renamed, added and split. Whatever you write today deserves a look every six months.
What about llms.txt?
You'll see llms.txt recommended a lot. It's a plain-text file at your site root that describes what your business is and where the important pages are, written for assistants rather than browsers.
Worth being straight about its status: it's a proposed convention, not a standard, and no assistant is obliged to read it. Nobody should promise you results from it.
I still put one on every site I build, for a reason that has nothing to do with crawlers. Writing a file that states plainly what you do, who you do it for and what it costs is a genuinely clarifying exercise, and the answers belong in your actual page copy too. If a machine picks it up, that's upside. If it doesn't, you've still written the clearest description of your business you own.
The related question — how a page earns a citation once a crawler is allowed in — is the subject of this piece on answer engine optimization, and this one on measuring whether it's working.