Google Clarifies Robots.txt Role In AI Training
Google's Gary Illyes explains robots.txt governs crawling, not AI training, and websites must target specific user-agent tokens to control AI access. This clarification matters for website owners who want to manage how their content is used.
Original reporting · searchenginejournal.com
Original: “Google’s Illyes: Robots.txt Covers Crawling, Not AI Training via @sejournal, @MattGSouthern”
Read “Google’s Illyes: Robots.txt Covers Crawling, Not AI Training via @sejournal,…” at Search Engine Journal →
How this was written. The 5:45 wrote this article with a language model from Search Engine Journal’s reporting (model: @cf/meta/llama-3.3-70b-instruct-fp8-fast); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from Search Engine Journal’s report
- Google's Gary Illyes clarified that robots.txt governs crawling, not AI training, and websites must target specific user-agent tokens to control AI access.
- Websites can use Disallow statements to prevent specific AI crawlers from accessing particular webpages, but there is currently no standard way to express a training preference.
- Google-Extended is a robots.txt token that manages whether content crawled by Google can be used to train future Gemini models, but its meaning comes from Google's documentation rather than RFC 9309.
- Percent encoding is recommended for non-English paths in robots.txt rules to avoid potential issues with older or non-Google crawlers.
- The IETF draft proposing a Content-Usage rule for robots.txt is still a Working Group Document and has not yet been submitted for review by the IETF's Steering Group.
Google's Gary Illyes stated in a Search Central video that robots.txt, as defined by RFC 9309, only governs which URLs a crawler may request, not what happens to the content afterward, according to Search Engine Journal. This means that robots.txt does not dictate what a system does with the data after it has already crawled it.
The video is part of The Non-Canonical Cabinet series, where Illyes and Cherry Prommawin of Google Search Relations address edge-case questions from the community. Illyes is also co-author of an IETF Internet-Draft that aims to give robots.txt a standard way to state usage preferences.
Illyes explained that to keep one AI crawler out while allowing others, websites must target that crawler's user-agent token in the robots.txt file. He also noted that if a website wants to completely block a specific AI crawler from accessing its entire website, it needs to explicitly target their specific user agent tokens in the file.
Google's robots.txt specification page states that only four elements are supported: User-Agent, Allow, Disallow, and Sitemap. Any invalid lines will be ignored by Google. Illyes illustrated a fictional directive that would be ignored by a parser, highlighting the importance of using valid and supported elements in robots.txt files.
Google-Extended is a robots.txt token that manages whether content crawled by Google can be used to train future Gemini models. However, this token is specific to Google and its products, and its meaning comes from Google's documentation rather than RFC 9309. Illyes co-wrote an IETF draft that proposes adding a Content-Usage rule to robots.txt, which would define preferences for how crawled content should be used.
Illyes' co-authored draft, dated August 19, would update RFC 9309 with a Content-Usage rule for robots.txt and a matching HTTP header. The draft is still a Working Group Document and has not yet been submitted for review by the IETF's Steering Group. The Working Group plans to send it to the IESG, but the current version is set to expire on February 20, 2027, unless a new revision replaces it sooner.
Cherry Prommawin advised websites to use percent-encoded versions of non-English paths in their robots.txt rules to avoid potential issues with older or non-Google crawlers. Google's parser handles raw characters by encoding them, but other crawlers might get confused by raw UTF-8 characters or emoji in paths.
The method for limiting AI use of website content while keeping the site visible in Google Search depends on the specific bot or product being used. Disallow statements may prevent specific AI crawlers from accessing particular webpages, but there is currently no standard way to express a training preference independent of the bots allowed to crawl the webpage.
Looking ahead, the next Non-Canonical Cabinet episode will cover the 404 status code. Website owners who want to manage how their content is used by AI models should stay informed about developments in robots.txt and usage preferences.
Search Engine Journal reported that websites using URLs with languages other than English should encode their robots.txt paths using percent encoding to ensure compatibility with different crawlers. This is because Google treats both raw and encoded versions of the same robots.txt rule path equally, but other crawlers could potentially interpret raw characters differently.
Questions and answers
What is the purpose of robots.txt?
Robots.txt governs which URLs a crawler may request, not what happens to the content afterward.
How can websites control AI access to their content?
Websites must target specific user-agent tokens in their robots.txt file to control AI access, and can use Disallow statements to prevent specific AI crawlers from accessing particular webpages.
What is Google-Extended and how does it relate to AI training?
Google-Extended is a robots.txt token that manages whether content crawled by Google can be used to train future Gemini models, but its meaning comes from Google's documentation rather than RFC 9309.
