A small file at the root of a website has become one of the most contested pieces of internet infrastructure. The robots.txt file began as a way to tell automated visitors which areas of a site they should avoid. In the era of generative AI, publishers are increasingly using crawler instructions to express a more complicated preference: whether their writing, images and other material should be used to train models.
OpenAI announced on October 30 that it will support a shared signal in robots.txt for expressing preferences about foundational generative-AI training. The idea is for publishers to send one broadly understood instruction rather than maintaining a separate rule for each participating AI provider.
There is one essential timing detail. OpenAI said it aims to implement the signal in the coming months. Its announcement is a commitment to a future change, not confirmation that the new shared setting is already being honored across its systems.
One preference is easier than a list of bot names
Today, a publisher may have to identify and manage individual crawlers. That creates operational work, particularly when providers change user agents, products and crawler behavior. The shared signal attempts to separate the underlying publisher preference from the constantly changing roster of companies operating AI systems.
If a common convention is adopted widely, a site owner could express a training-related choice once and expect participating providers to interpret it consistently. That would not necessarily end the need for provider-specific rules, but it could make the baseline simpler.
OpenAI said the effort is voluntary and connected to work by the Internet Engineering Task Force’s AI Preferences working group. The working group context is important: a proposed shared practice is not the same thing as a universally enforced legal right or a completed technical standard.
Robots rules do not settle the copyright debate
A website preference signal tells a compliant automated system what an operator wants. It does not automatically determine copyright ownership, lawful access, contractual permissions or how every other company may use publicly available material.
There are also different uses of web content to distinguish. A provider might crawl pages for search indexing, retrieve them to answer a current user query, or use collected material in foundational model training. Publishers can have different preferences for each activity.
OpenAI’s announcement concerns training preferences. It does not say the shared signal will prohibit all automated access, nor does it announce a new industry-wide rule for how AI answers may cite web pages. Conflating those functions would lead publishers to expect protections that have not been described.
Existing controls remain the practical tool today
OpenAI said its existing crawler controls will continue to apply and be respected. That is the actionable point for website operators now. Until the new signal is implemented and documented, they should continue to use the controls that are already available rather than assuming a future shared rule has replaced them.
Operators should also watch for implementation documentation explaining the exact syntax, what training activities are covered and whether the signal applies prospectively. A future preference setting should not be presumed to reverse uses of content that have already occurred.
The real test is participation beyond one company
The administrative burden disappears only when a significant number of AI providers adopt compatible behavior. If different companies interpret a common instruction differently, publishers will still need a matrix of exceptions and verification tools.
Compliance also needs some degree of observability. A website can publish a crawler preference, but it may not be able to verify independently whether that preference influenced a downstream training dataset. Public commitments, technical documentation and external accountability will therefore matter alongside the specification itself.
OpenAI has backed a potentially useful simplification for website operators. What is live today is the commitment and the existing crawler-control system. The promise of a common training preference becomes meaningful only when more providers implement it, describe its boundaries clearly and follow it in practice.
Primary sources
OpenAI, making it easier for websites to express AI training preferences (October 30, 2026)
IETF AI Preferences working group reference linked by OpenAI
