Website owners who want to keep their pages out of AI training currently write a separate rule for each company: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google. A proposal published on 30 October would add a shared AI training opt-out to robots.txt, the plain text file that tells crawlers which pages they may visit. One group of lines could then reach several providers.
OpenAI, Google and Microsoft say they plan to honor it. Anthropic, where the proposal’s author works, says it has started building support. None of them reads the signal everywhere today, and each is adopting it on its own schedule and with its own exceptions.
For most publishers, the practical worry is search traffic. Refusing training does not have to cost visibility, because most providers control training separately from search, although Google’s Google-Extended combines Gemini training and grounding choices. That still means picking the right setting for every provider, and remembering that crawlers obey robots.txt voluntarily.
What the shared AI training rule does
Kevin Kelley of Anthropic wrote the proposal, an Internet-Draft titled “Machine-Readable Training Preference Signal”. It is an individual submission to the Internet Engineering Task Force (IETF), the group that develops many Internet standards, and the IETF has not endorsed the shared training preference draft.
This draft version expires on 3 May 2027. That is not a provider implementation deadline.
It reuses ordinary robots.txt syntax. A site adds a group that begins User-Agent: __usage_ai-training and lists normal Allow and Disallow lines under it. Disallow: / refuses training on the whole site, and the same lines can exclude one directory or allow only a press kit.
Training has a narrow meaning here. The draft covers updates to the parameters of a generative model, meaning the internal values it learns from data, and excludes systems used only for classification, ranking, or scoring. A preference attaches to content when it is retrieved, so the draft says nothing about pages collected before the signal existed.
Two rules decide what most sites are saying right now. A User-Agent: * group is the fallback for crawlers without a matching group, but it never supplies this shared preference. Without __usage_ai-training, no shared training preference is expressed; existing provider controls still apply.
InfoSeely’s own robots.txt shows what that means. On 31 October, InfoSeely’s robots.txt contained one wildcard group, an empty Disallow line, and a sitemap link. It imposed no crawl restrictions and expressed no shared training preference. Any site that has never added AI-specific lines is in the same position.
Which AI companies will honor it and when
Their plans range from a fixed start date to an open-ended promise.
| Company | What it has said | Timing |
|---|---|---|
| OpenAI | Aims to implement the signal | “In the coming months”, no date |
| Aims to support it along with other AI providers | 2027, with crawler documentation saying early 2027 | |
| Microsoft | Will honor the new preferences as it crawls | From 1 February 2027 |
| Anthropic | Has started implementing support | No completion date |
OpenAI’s implementation plan gives no date, and Anthropic has begun implementation without saying when it will finish. The announcements give no adoption count.
When older provider rules still win
Providers may give their own usage controls priority, and private agreements may override the shared preference. Crawler access rules apply independently. One file can therefore mean different things to different companies.
Google has published the clearest rule. Under Google’s explicit exception, it plans to treat a shared opt-out as a Google-Extended refusal only when the site has no Google-Extended line of its own. Take this file:
User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/ User-Agent: Google-Extended Allow: /
Providers that follow only the shared group would keep article pages out of training and allow the press kit. Google would follow the explicit Google-Extended permission for Gemini training and grounding. Grounding supplies web content to an AI model when it answers. This does not change crawler access. That reading comes from the published rules, since Google has not switched the control on.
Microsoft splits training from grounding. Training can be set site-wide in robots.txt or per page, and the page setting wins. Grounding is set only per page: allow, disallow, or allow only Microsoft’s own experiences. Both default to allow. Once Microsoft finds either new setting, it ignores the site’s older NoArchive and NoCache tags, so adding only a training rule could drop grounding limits those tags provided. From 1 March 2027, those tags will no longer express Microsoft training or grounding preferences. Sites that rely on them should follow Microsoft’s transition requirements.
Both OpenAI and Anthropic say their existing crawler controls stay in place. Their announcements do not fully specify how overlapping training preferences will be resolved.
Blocking training is not the same as leaving search
A site can refuse AI training and still be found in search and AI answers. This sample shows one way to do it:
User-Agent: __usage_ai-training Disallow: / User-Agent: GPTBot Disallow: / User-Agent: ClaudeBot Disallow: / User-Agent: Google-Extended Disallow: /
The shared group asks every provider that reads the signal not to train on the site. The other three groups carry the same request in each company’s current format:
- GPTBot is OpenAI’s training crawler. OpenAI’s documentation says “Each setting is independent of the others”, so OAI-SearchBot can still index the site for ChatGPT search.
- ClaudeBot collects content Anthropic may use for training. Claude-SearchBot, which supports Claude’s search results, is separate.
- Google-Extended covers Gemini training, grounding in Gemini Apps and Grounding with Google Search on Vertex AI, Google’s cloud AI platform. Google’s existing training and grounding control does not govern Google Search or AI Overviews.
This standalone sample does not block Googlebot, Bingbot, OAI-SearchBot or Claude-SearchBot. Other robots.txt rules, access restrictions and product settings still affect search and citation eligibility. AI Overviews and AI Mode have their own switch: a “Search generative AI” setting in Search Console, which Google describes as controlling “your site’s visibility in generative AI features on Google Search, such as AI overviews and AI mode.” It was set to Include on InfoSeely’s account.
What the site gives up is Gemini Apps. Google says sites that specifically allow Google-Extended “are eligible to appear in responses and receive clicks via grounding in Gemini Apps”, so refusing it gives that up. Google plans the same treatment for the shared refusal when Google-Extended is not explicitly specified.
Microsoft’s announcement does not name a separate training crawler. From 1 February 2027, it will read a site’s training choice from the shared group or from page-level settings. Grounding stays allowed unless individual pages say otherwise.
Until the shared token is honored everywhere, the provider-specific lines do the real work. User-directed fetchers are a different case: ChatGPT-User and Claude-User visit pages when a person asks, and OpenAI says robots.txt may not apply to ChatGPT-User. In our GPT-6.1 Terra test, ChatGPT’s agent identified itself to the test site through signed request headers naming chatgpt.com. Treat the file above as a sample to adapt. It is not legal advice, and providers may change their rules.
What website owners should keep for now
The shared group can be published now, but implementation is still progressing. Keep existing GPTBot, ClaudeBot, and Google-Extended controls alongside it.
At Google, the decision needs more thought. Weigh Gemini Apps citations against keeping pages out of Gemini training, then state the choice in an explicit Google-Extended line, because that line overrides the shared group.
Sites that use NoArchive or NoCache to limit Microsoft have the tightest timetable. Microsoft starts honoring the new settings on 1 February 2027, and before 1 March 2027 those sites should publish training and page-level grounding settings. Sites using the old tags on 15 December 2026 can also record a temporary grounding preference in Bing Webmaster Tools, which lasts until the page is recrawled or 30 September 2027, whichever comes first.
Sufyan’s view: InfoSeely has decided to refuse AI training while keeping search crawlers open, because our tests take real time and we want that work to reach readers. We will add the shared token and the provider-specific lines, accepting that our pages leave Gemini Apps grounding. Before saving a file like ours, check that no other rule blocks the search crawlers you want.
