Most people asking whether to block AI are asking one question when there are three. Can an AI company read your site? Can it quote you in an answer and send someone to you? Can it use your words as training data? Those are separate, and the crawlers are already separated so you can say yes to some and no to others. Our own answer is yes, yes and no.
This is the position we took on doparagroup.com, why, and what the numbers actually show. You can read the file itself at doparagroup.com/robots.txt — the reasoning is written into it.
The three questions, and why they get muddled
Crawling is a machine fetching your pages. Citing is an assistant quoting or linking you when someone asks a question. Training is your words becoming part of a model's weights. People collapse all three into "letting AI have my content", then block everything and quietly remove themselves from a growing slice of how buyers find suppliers.
The companies themselves have split the difference already. Anthropic runs three separate crawlers — one for training, one for building a search index, one for fetching a page because a user asked about it right now. OpenAI runs GPTBot for training and OAI-SearchBot for the ChatGPT search index. They are individually controllable by name, which means a blanket block is a decision you almost certainly did not intend to make.
The number that makes people want to block
AI crawlers take far more than they send back. Cloudflare's own measurements, pulled on 1 August 2026, put the crawl-to-referral ratio at 1,917 pages crawled per referral for Anthropic's crawler, 251 for OpenAI's, and 289 for Perplexity's. For comparison, a traditional search crawler is orders of magnitude better behaved on that measure.
Publishers have noticed. In July 2026 around 9.6% of AI bot requests were being refused outright, rising to nearly 13% in the last week of the month, and GPTBot appears in more blocking rules than any other AI crawler.
That is a real cost and a fair grievance if you are a large publisher whose business is traffic. For a firm whose website exists to start conversations, the calculation is different: you are not selling page views, and one buyer arriving from an AI answer is worth more than the bandwidth of a thousand crawls.
What we decided
Our robots.txt allows every named AI crawler, and carries a Content Signal declaring three things separately:
- search=yes — index us and link to us. Obviously yes.
- ai-input=yes — ground AI answers in this content. Being the quoted source in an answer is a real route to a first conversation.
- ai-train=no — we are happy to be cited, not to be training data.
The reasoning for the third one is the part worth stealing: declining to be training data costs us nothing, because training does not send anyone here. Being quoted does. Those two are worth entirely different amounts to a small firm, so it makes no sense to give the same answer to both.
Being straight about the limits: a Content Signal is a stated preference, not an enforcement mechanism. It records a position clearly, in public, in the place a crawler looks. It does not stop anyone. If you need enforcement rather than a statement, that is a different job involving your host or CDN.
llms.txt, and why we expect nothing from it
We publish an llms.txt — a plain-text summary of the site aimed at language models. We also think it is very unlikely to do anything for us, and we would rather say so than sell it.
As of June 2026 only 8.7% of the top thousand websites publish one. Nearly half of those thousand are infrastructure domains that cannot host a file at their root at all; among the 549 that can, adoption is 15.8%. No major AI vendor has committed to reading it in production. It costs us one command to generate and it is honest and tidy, so it stays — but if an agency quotes you for "AEO optimisation" and the deliverable is an llms.txt file, you are buying a text file.
What actually gets you into AI answers
The same thing that has always got people quoted: being the only place a specific, checkable fact is written down.
Assistants summarising "what does a bespoke database cost in the UK" need a number from somewhere. A page that publishes an actual price, an actual timescale, or an actual account of a project is quotable. A page that says "costs vary depending on your requirements — contact us for a quote" is not, and never will be, no matter what is in its robots.txt.
That is why our own guides carry real figures and, where the honest answer is unflattering, say so. It is not a growth tactic. It is just that unfalsifiable copy cannot be cited by anything.
What we would actually do, in your position
- Do not blanket-block. If you have already, check what you blocked — you very likely refused the search crawlers alongside the training ones.
- Decide the three questions separately, and write the answers into robots.txt where anyone can read them.
- Allow the ones that can send you people: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Google-Extended and the user-initiated fetchers.
- Say no to training if you want to — it costs you nothing in traffic.
- Then go and write something quotable, which matters more than every switch above combined.
If you want a second opinion on what your own site currently allows, send us the address — reading a robots.txt takes a minute and we will tell you what it actually says rather than what you think it says.