Today, opting out of AI training means one robots.txt rule per AI company. Google has announced it plans to support one shared rule instead.
What changed
On October 30, 2026, Google published Supporting a shared AI training control for websites on the Search Central blog. Google says websites will be able to use a shared "policy token" in robots.txt "to set broad preferences across participating AI providers." It is based on work from the AIPREF working group at the Internet Engineering Task Force (IETF).
Google's crawler documentation, updated the same day, gives the rule: user-agent: __usage_ai-training. The token comes from an IETF Internet-Draft, Machine-Readable Training Preference Signal. It is a first draft (version -00), not a finished standard. Its example looks like this:
User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/
The draft says this expresses a preference against AI training for the whole site, except /press-kit/, where it expresses a preference in favor.
How Google will read it
- It maps to Google-Extended. Google says that when a site opts out with the token "(and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended."
- Named rules still work. Google says "Existing company-specific controls will continue to function alongside this new joint signal." So you can opt out of everyone with the token, then set a different rule for Google-Extended.
- It is not live yet. The blog post says Google aims to support it "along with other AI providers in 2027." The documentation says "early 2027."
What it does not do
- It does not affect Google Search. Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search," the documentation says.
- It does not block crawling. The draft says the token "is not a directive for crawling behavior." To stop a crawler, you still need a rule for that crawler's own name.
- User-agent: * does not count. The draft says wildcard groups must not be read as matching the token. If your file has no __usage_ai-training line, no preference is expressed.
One trade-off to know
Google-Extended covers more than training. It controls whether Google may use your content to train Gemini models and for grounding in Gemini Apps and Vertex AI. Google's blog post says "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."
So, on Google's description, opting out with the shared token alone is read as disallowing Google-Extended. If you want to stay out of training elsewhere but still be eligible for Gemini Apps grounding, you would need an explicit Google-Extended rule that allows it.
What to do now
- Decide your stance on AI training. It is your choice, and it does not change your Google Search rankings.
- Check your robots.txt for Google-Extended, GPTBot and ClaudeBot. These named rules are the ones that work today.
- If you add the token, give it its own group. Don't add it to a group that also names Googlebot or Bingbot. A Disallow: / in that group would block those search crawlers too.
- Don't expect an effect yet. Google plans support in 2027. Until then, the token does nothing at Google.
- Remember that Google-Extended has no separate user agent string. You won't see it in your server logs. Google crawls with its usual user agents and reads the token as a control.
See which bots your robots.txt lets in
The SEOSFT AI search check reads your robots.txt and shows, bot by bot, whether Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, GPTBot, ClaudeBot and Google-Extended are allowed. Training bots are reported only and don't change your score. It does not check the new __usage_ai-training token yet. For bot names, see our guide Which AI crawlers to allow in robots.txt.
Sources
- Google Search Central Blog, Supporting a shared AI training control for websites (Oct 30, 2026): https://developers.google.com/search/blog/2026/10/ai-training-control
- Google, Google's common crawlers: Google-Extended (last updated Oct 30, 2026): https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended
- IETF Internet-Draft, Machine-Readable Training Preference Signal, draft-kelley-aipref-training-preference-signal-00 (Oct 30, 2026): https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/