RFC 9309 Specification: How Bots Parse Directives

In September 2022, the Internet Engineering Task Force (IETF) ratified RFC 9309 as the official proposed standard for the Robots Exclusion Protocol (REP). Prior to RFC 9309, robots.txt interpretation was informal, causing severe discrepancies between crawlers.

Key RFC 9309 Precedence Mechanics:

  1. Specific User-Agent Precedence: A bot must only obey the record that matches its exact user-agent name. If no exact match exists, it falls back to the wildcard User-agent: * group. It does not merge directives across groups.
  2. Longest Path Match Wins: If multiple directives in the matched group match the target URI, the rule with the longest path string takes absolute priority (e.g. Allow: /articles/ with length 10 beats Disallow: / with length 1).
  3. Allow Beats Disallow on Equal Length: If both an Allow and Disallow rule match the same target URI and have identical character lengths, the Allow directive wins.
  4. Default Permissive State: If no rule matches a requested URI path, crawlers assume access is permitted by default.

Why Blocking "Google-Extended" Does Not Hurt Google Search

Many publishers fear blocking AI crawlers will destroy their organic search traffic. Google maintains strict separation between their search indexer and AI training crawlers:

  • Googlebot: The primary crawler powering Google Search indexing, PageSpeed metrics, and SERP snippets. Disallowing Googlebot will deindex your site.
  • Google-Extended: A standalone crawler token used specifically to train Gemini models and Vertex AI foundation weights. Blocking Google-Extended has zero impact on your Google Search rankings or indexing.

Frequently Asked Questions

Does robots.txt legally prevent AI companies from training on my data?
Robots.txt is a voluntary technical standard. Legitimate AI laboratories (OpenAI, Anthropic, Google, Cohere, Apple) program their crawlers to strictly honor RFC 9309 directives. Rogue scrapers may ignore robots.txt and require Cloudflare WAF bot management or IP rate limiting.
What is the difference between GPTBot and ChatGPT-User?
GPTBot crawls the open web asynchronously to build pre-training datasets for future GPT models. ChatGPT-User is triggered only when an active user in ChatGPT asks the model to browse a specific link in real time.
Can I block AI bots on specific directories only?
Yes. You can permit AI bots on marketing pages while blocking proprietary documentation or premium content by specifying: Disallow: /members/ or Disallow: /api/.