OpenAI will honor one robots.txt signal for AI training preferences

OpenAI says it will honor a shared signal in robots.txt that lets website owners state whether their content may be used to train foundational generative AI models. The company plans to implement the signal in the coming months.

Today, publishers often have to identify individual AI crawlers and maintain separate rules for each provider. A common preference should reduce that work by letting one declaration communicate the same choice to every participating service. OpenAI says its existing crawler-specific controls will continue to apply, so the new mechanism will add to rather than replace them.

The proposal builds on work by the Internet Engineering Task Force’s AI Preferences group. It is voluntary, and OpenAI is encouraging other AI companies to support it. That means it is not yet a universal enforcement system, and implementation details still matter.

For publishers, the practical benefit is simpler control over training use without having to block every AI crawler individually. For the AI industry, the move is a step toward a common language for content rights on the open web—an area that has so far relied on fragmented technical rules and provider-specific policies.