Google-Extended and the New Shared AI-Training Opt-Out: What Each Control Blocks (and What It Doesn't)
Table of Contents
What did Google, OpenAI and Microsoft announce on October 30?
What does each control block, and what doesn't it?
Robots.txt audit: what AI-writing sites actually publish (Nov 6, 2026)
What should StealthGPT do?
FAQ
On October 30, 2026, Google, OpenAI and Microsoft each said they would honor a shared AI-training preference in robots.txt. The idea is simple enough: one signal instead of a separate rule for every crawler. The hard part is reading what each control actually blocks, when it starts mattering, and what peer sites have done with the file so far.
This post is not a rewrite of the three announcements. It pairs them with a dated robots.txt audit of AI-writing and detector sites, fetched on November 6, 2026, and with an open product question for StealthGPT's own file. The shared token is __usage_ai-training. Spell it exactly that way.
What did Google, OpenAI and Microsoft announce on October 30?
All three posts land on the same day and point at the same IETF AIPREF work. Google's Search Central blog, Supporting a shared AI training control for websites, says website owners will be able to use a shared policy token in robots.txt to set broad preferences across participating AI providers, building on Google-Extended, which Google has offered since 2023.
OpenAI's post, Making it easier for websites to express AI training preferences, says OpenAI will honor a common signal in robots.txt for foundational generative AI training preferences, aims to implement it "in the coming months," and keeps its existing crawler controls in place.
Microsoft's Bing post, Your content, your choice: clearer preferences for training and grounding, documents the shared robots.txt user-agent as __usage_ai-training, adds a separate page-level AI Grounding preference, and says the new prefs do not change eligibility for traditional Bing search results.
None of the three posts says the shared token, by itself, removes a site from classic search. Company-specific controls stay alongside it. Timing differs by vendor, which matters more than the press-release framing.
What does each control block, and what doesn't it?
Google-Extended (existing, since 2023)
From Google's October 30 post: Google-Extended is the AI training control Google has offered since 2023. The shared-token post says the new control, when used to opt out and when Google-Extended is not explicitly specified, is something Google "plans to interpret… as if the publisher had disallowed Google-Extended." Sites that "specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." Google aims to support the new shared control "in 2027."
What the October 30 post does not claim: that Google-Extended is a Search ranking signal, or that opting out removes you from Google Search. Read it as a training and Gemini Apps / Vertex grounding control, not as a substitute for snippet or indexing controls. For measuring AI Overview clicks in Search Console after Google's October 15 link changes, see our separate post on that report rather than stretching Google-Extended to cover it.
(Measurement post: AI Overviews Clicks in Search Console: How to Measure the October 15 Link Update.)
Shared token __usage_ai-training
Microsoft's robots.txt example for a whole-site training opt-out is:
User-Agent: __usage_ai-training
Disallow: /
Allow is the default when no applicable training preference is published. OpenAI says it will honor the common signal and aims to implement it in the coming months, without replacing its existing crawler rules. Google aims for 2027 alongside other providers. Do not read any of those timelines as "already live everywhere."
Microsoft training vs grounding (separate)
Microsoft splits the choice. AI Training can be set in robots.txt or at page level (Allow or Disallow; Allow is default). AI Grounding is page-level only: Allow (default), DisallowExtended, or Disallow. DisallowExtended limits grounding to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot, and Microsoft 365, and excludes third-party grounding services Microsoft names (Web IQ, Foundry API grounding).
Dates from the Bing post: new prefs honored as crawled from February 1, 2027. Sites on NoArchive/NoCache should migrate by March 1, 2027. Temporary grounding prefs in Bing Webmaster Tools expire September 30, 2027. Again, these prefs "do not change eligibility for traditional Bing search results."
| Vendor | When the new shared / related prefs matter | What stays true |
|---|---|---|
| OpenAI | Aims to implement shared signal "in the coming months" | Existing crawler controls remain |
| Aims to support shared control in 2027 | Google-Extended remains; Search eligibility not framed as the point of this control | |
| Microsoft | Honors new prefs as crawled from Feb 1, 2027; NoArchive/NoCache migration by Mar 1, 2027 | Training and grounding are independent; traditional Bing results unchanged by these prefs |
Robots.txt audit: what AI-writing sites actually publish (Nov 6, 2026)
Method, from the outline's dated audit: each site's public /robots.txt was fetched over HTTPS on 2026-11-06 (about 9:29 AM America/New_York / box UTC ~14:29). The check looked for __usage_ai-training, Google-Extended, GPTBot, ClaudeBot/anthropic, CCBot, Bytespider, and OAI-SearchBot. Quillbot and Turnitin returned HTTP 403; those rows are unread fetches, not "no rules."
Headline: of the AI-writing and detector sites in the set, none had adopted the shared __usage_ai-training token yet. Most also had no Google-Extended or GPTBot group at all, which for vendors that treat absence as allow means those products are not blocked by name.
| Site | Fetched | __usage_ai-training | Google-Extended | GPTBot | Notes |
|---|---|---|---|---|---|
| stealthgpt.ai | 200 | absent | absent | absent | User-Agent: * Allow /; Disallow /account |
| undetectable.ai | 200 | absent | absent | absent | Allow-heavy * |
| humanize.ai | 200 | absent | absent | absent | User-Agent: * Disallow empty |
| undetectedgpt.com | 200 | absent | absent | absent | Allow / |
| bypassgpt.ai | 200 | absent | absent | absent | Allow / |
| originality.ai | 200 | absent | absent | absent | Disallow /popup only |
| gptzero.me | 200 | absent | absent | absent | Allow / |
| pangram.com | 200 | absent | absent | absent | No AI-training groups |
| winston.ai | 200 | absent | absent | absent | Allow / |
| zerogpt.com | 200 | absent | absent | absent | Allows OAI-SearchBot |
| surfer.ai | 200 | absent | absent | absent | Disallow /dev/ |
| jasper.ai | 200 | absent | Allow: / | Allow: / | Explicitly allows Google-Extended, GPTBot, ClaudeBot, CCBot; Disallows Omgilibot |
| phrasly.ai | 200 | absent | Allow: / | Allow: / | Comment: blog written to be cited; allows ClaudeBot, PerplexityBot, Applebot-Extended too |
| copy.ai | 200 | absent | absent | absent | Path Disallows under * |
| writesonic.com | 200 | absent | absent | absent | Allow / |
| hix.ai | 200 | absent | absent | absent | App/share path Disallows |
| grammarly.com | 200 | absent | absent | absent | Path Disallows under * |
| scribbr.com | 200 | absent | absent | path Disallow for ClaudeBot folders | Not a full ClaudeBot block |
| quillbot.com | 403 | — | — | — | Could not read |
| turnitin.com | 403 | — | — | — | Could not read |
Spot-checks on November 6, 2026, matched the audit on the three files that matter most for this post: stealthgpt.ai still allows everything under User-Agent: * except /account; jasper.ai still publishes explicit Allow: / for Google-Extended and GPTBot; phrasly.ai still allows Google-Extended and GPTBot and comments that the blog is written to be cited.
Takeaway: roughly five weeks after the October 30 announcements, the shared token has almost no adoption on this peer set. A few AI-writing brands already allow Google-Extended and GPTBot on purpose. Most competitors simply never named those agents.
What should StealthGPT do?
Current state, fetched November 6, 2026: StealthGPT's robots.txt allows everything under User-Agent: * except /account. It does not set Google-Extended, GPTBot, or __usage_ai-training.
That is a fact about the file. It is not a public company stance about AI training. Before this post goes live, product needs to pick one of the following and date it:
Stay open (current): keep allowing AI training/grounding crawlers; optionally add explicit Allow: / groups for Google-Extended / GPTBot / ClaudeBot like Phrasly, if citation in AI answers is the goal.
Opt out of training via the shared token once Google/OpenAI/Microsoft honor it, and/or add User-agent: Google-Extended with Disallow: / now.
Split path rules (for example allow blog, disallow app) — only if product agrees.
Until that decision is dated, this draft reports the current file and leaves the choice as a pre-publish gate. Do not invent a StealthGPT robots.txt policy in the meantime.
If the open question for you is whether Google treats AI-assisted pages as spam, that is a different control surface entirely. See Does Google Penalize AI Content in 2026? and Does Programmatic SEO Count as AI Slop?. For drafting structured posts you still review before publish, StealthGPT's SEO Writer is the product page for that workflow.
FAQ
What is Google-Extended?
A robots.txt product token Google has offered since 2023. Google's October 30, 2026 post treats it as the existing AI training control that the new shared token will sit beside, including Gemini Apps grounding eligibility when Google-Extended is specifically allowed. It is not framed there as a Search ranking signal.
What is __usage_ai-training?
The shared robots.txt user-agent token for AI-training preferences that Google, OpenAI and Microsoft announced support for on October 30, 2026. Microsoft documents User-Agent: __usage_ai-training with Disallow: / to opt out.
Does opting out of AI training remove you from Google or Bing search?
Not according to these vendors' October 30 posts. Google presents company-specific and shared controls alongside Search. Microsoft says the new prefs do not change eligibility for traditional Bing results.
When do the new rules take effect?
OpenAI: "coming months." Google: aims for 2027. Microsoft: honors new prefs as crawled from February 1, 2027, with a March 1, 2027 migration deadline for NoArchive/NoCache users.
Have AI-writing sites adopted the shared token yet?
In a November 6, 2026 fetch of about 20 peer robots.txt files, none included __usage_ai-training. A few already allow Google-Extended and GPTBot explicitly.
