One robots.txt Rule for AI Training: What Builders Should Do Now
Google, Microsoft, OpenAI and Anthropic back one AI training opt-out. See training versus grounding, and what builders should do.
Bilal
5 min read
On 30 October 2026, the biggest names in search and AI backed one shared robots.txt rule. It lets a website opt out of AI training. Google, Microsoft, OpenAI and Anthropic will honour it, according to PPC Land.
So should a builder change anything? This guide explains the news in plain English. It also shows a sensible default. First, a note. Dates may shift, so check each company’s own documentation before you act.
What Was Announced
- A shared AI training preference now works through robots.txt, using a policy token called __usage_ai-training.
- Then you list the paths you do not want used for training. A site can block everything, or everything except one folder.
- PPC Land says the signal comes from an IETF draft by Anthropic’s Kevin Kelley.
- Microsoft will honour the preferences from 1 February 2027, and Google aims to support the token in early 2027.
Microsoft’s Krishna Madhavan also announced separate Bing controls. They keep AI training apart from AI answers.
Training and Grounding in Plain English
These two words matter, because they do different jobs.
- Training uses your content to teach an AI model. The model learns from it once, then keeps that knowledge.
- Grounding uses your page when an AI writes an answer. The answer can cite or link back to you.
Microsoft now treats them as independent choices. For example, you can disallow training and still allow grounding. Microsoft also offers a limited option. It lets your content support Microsoft’s own AI products, but not its third-party grounding services.
Dates to Know
- 15 December 2026: temporary grounding controls open in Bing Webmaster Tools for sites that use the old NoArchive or NoCache tags.
- 1 February 2027: Microsoft starts honouring the new preferences as it crawls.
- 1 March 2027: the old tags stop counting as AI preferences.
- 30 September 2027: the temporary grounding preferences expire.
Most builder websites never used NoArchive or NoCache. So the transition dates probably do not affect you. Ask your developer to confirm.
What It Means for Builders
Microsoft warns that blocking grounding will significantly reduce your visibility in AI experiences. It does not change your place in normal Bing results. It also does not create any licensing or payment deal.
So the trade-off is simple. Blocking training is a choice about control. Blocking grounding costs you AI visibility. For most builders, the second choice makes little sense, because AI answers can mention your firm. Our guide to LLM optimization explains how.
A sensible default looks like this:
- Allow grounding, so AI answers can cite your pages.
- Decide on training separately, based on how you feel about your content.
- Keep all your normal search crawlers allowed.
However, remember our earlier caution. AI search sends far less traffic than Google today. See our post on whether AI search is worth it. So this is a housekeeping decision, not an emergency.
A Short Checklist
- Open your robots.txt file at yoursite.com/robots.txt.
- Note which AI crawlers it allows or blocks today.
- Then decide whether you want to opt out of training.
- Also check your page source for NoArchive or NoCache tags.
- Create a free Bing Webmaster Tools account, if you have none.
- Look at its AI Performance report to see whether AI answers cite your site.
- Write down your decision, and review it in the new year.
What Not to Do
- Never block every bot in a panic. You could hide your site from normal search.
- Avoid copying a robots.txt file from another site, because it may block pages you need.
- Skip any plugin that promises to ‘protect’ your content without explaining the rules.
- Do not expect payment, because the new preferences do not set up licensing deals.
Why Bing Matters Here
Also, some AI tools lean on Bing for live answers. We explained this in our LLM optimization guide. So Bing Webmaster Tools deserves a few minutes of your time. Its AI Performance report shows citations, cited URLs and grounding queries across supported AI experiences, according to Search Engine Roundtable. That makes it a useful free check.
Also keep your business details consistent on Bing Places. Our guide to Google Business Profile changes covers the profile side.
How We Can Help
We can read your robots.txt file, explain each rule in plain English and set a sensible AI policy. We can also set up Bing Webmaster Tools and check your AI citations. See our SEO for construction companies service.
Book a free consultation at bilalwebstudio.co.uk or email [email protected].
Frequently Asked Questions
Do I have to change my robots.txt file?
No. The new preferences are optional. Change your file only if you want to opt out of AI training.
What is the difference between AI training and AI grounding?
Training teaches a model with your content. Grounding uses your page to write a live answer, often with a link back to you.
Will blocking AI training hurt my Google rankings?
The announcement says the preferences do not change eligibility for normal Bing search results. Google plans to support the shared token in early 2027, so check its documentation then.
Should a builder block AI grounding?
Usually not. Microsoft says blocking grounding will significantly reduce your visibility in AI experiences. Most builders want AI answers to cite them.
