One robots.txt Line Now Says No to AI Training. Nobody Has to Read It.

A site owner who wants out of AI training has been writing the same Disallow block once for every crawler that shows up. An IETF draft published this week replaces that pile with one product token.

The document is draft-kelley-aipref-training-preference-signal-00, from the IETF AI Preferences working group. It adds a policy token, __usage_ai-training, written inside a normal robots.txt group. That is the whole invention, and the robots.txt training signal is the first shared token of its kind. The draft says explicitly that it introduces no new directive names and no new grammar, and that the path language, group structure and longest match precedence from RFC 9309 are unchanged.

The token says nothing about crawling

This is the part publishers get wrong. A policy token records how you want content used. It is not a crawl directive. A crawler still checks the group named for its own product token, and those rules still decide whether it fetches the page.

The draft permits a consumer to read a preference against training and decide not to fetch. Permits, not requires. Content usage and traffic management stay orthogonal, which means adding the token is not a substitute for blocking a bot you do not want in the first place.

The wildcard block you already wrote does not count

Here is the trap. The draft says a wildcard group, the User-agent line with an asterisk, must not be interpreted as matching the policy token. A site with one blanket block and nothing else has expressed no training preference under this mechanism at all.

Silence is just as ambiguous on the other side. No group carrying the token means no preference, a vendor specific token from a previous era does not count, and where two expressions conflict the draft leans on specificity, then on private agreements, which may override the file entirely. Read the working group vocabulary draft for what counts as training. It means modifying the learned parameters of a generative model, and it excludes models used only for non generative tasks.

Your logs will not tell you who obeyed

The token never has to appear in a request header. The draft says a policy token need not appear in the User-Agent field of any outbound request. A crawler can honor your preference and look identical in the access log to one that ignored it. There is no audit trail, only trust.

SiliconANGLE reported that Google, Microsoft, Anthropic and OpenAI back the signal, with OpenAI and Microsoft implementing in the coming months and Google next year. OpenAI’s crawler documentation still lists the per bot controls, and the announcement said those keep applying alongside the shared signal.

For an operator, that is a preference channel, not a control. If a license term matters, it belongs in a contract. If bandwidth matters, the rate limits and the WAF are still doing the work. The same gap between a stated policy and an enforced one showed up in the traffic Wikimedia absorbed from agents that never announced themselves.

Three questions before you edit the file

Does your robots.txt contain a group that names the training token, or only wildcards that now express nothing? Does anyone in your organization own the difference between blocking a crawler and blocking a use? And if a vendor says it honors the signal, what evidence would you accept?

Write the group anyway. It costs one block, it is exactly backward compatible, and it is the only machine readable position a site has. Just do not confuse a preference with an enforcement mechanism, because the draft is honest enough to say which one it built.