Robots.txt Won’t Stop AI Training – What Small Biz Must Know

What Happened

In a recent episode of Google’s The Non-Canonical Cabinet, Search Quality Engineer Gary Illyes discussed the role of the robots.txt file (RFC 9309) in web standards. He explained that this file controls which URLs a crawler can request, but it does not dictate how that content is used once it has been fetched. Essentially, while blocking a bot through robots.txt prevents it from crawling your pages, it does not stop that bot—or any AI system—from utilizing previously collected content for large-language-model (LLM) training.

Illyes stressed that if a website aims to exclude a particular AI crawler while still permitting Google Search access, the only dependable approach is to target the crawler’s specific user-agent token within the robots.txt file. He also mentioned an upcoming IETF Internet-Draft, co-authored with Mozilla’s Martin Thomson, which aims to provide robots.txt with a formal mechanism to express usage preferences (e.g., indicating “don’t use for AI training”).

This conversation fits into a broader examination of nuanced SEO issues and highlights the increasing friction between established web-crawling protocols and the evolving landscape of AI-driven content consumption.

---

Key Details

• Robots.txt scope: Only controls crawling (URL fetching) and does not address post-fetch usage, such as AI model training.
• User-agent targeting: To block a specific AI bot, you must specify its exact user-agent string in the robots.txt file.
• IETF draft: A proposed extension could allow sites to implement a X-Robots-Tag: noai-style directive, but this is not yet an established standard.
• Non-English paths: Google recommends percent-encoding non-English rule paths for enhanced compatibility.
• Practical impact: Even if you block Googlebot, other crawlers (including open-source or proprietary AI bots) can still access and ingest publicly available content.

---

What It Means For Your Business

1. **Your Content Can Still Power AI Tools**

Even with a tightly controlled robots.txt, the information you publish could still be harvested by AI services that have already indexed it or that operate outside standard crawling rules. These services drive tools like ChatGPT, Perplexity, Gemini, and Google AI Overviews—the very platforms your customers turn to for discovering local businesses.

2. **Citation & Attribution Become Critical**

When AI models reference or paraphrase your content, they often fail to provide a direct link back to your site. VisibilityAI assists you in embedding structured data, schema markup, and citation-friendly snippets, enhancing the likelihood that an AI will present a click-through link instead of merely a text response.

3. **Re-evaluate Your SEO Playbook**

While traditional SEO practices (like meta tags and sitemap submissions) remain essential, you must now also:

• Publish high-quality, unique content that AI models find valuable enough to reference with a source link.
• Utilize schema.org (e.g., LocalBusiness, Product, FAQPage) to provide AI with clear signals regarding the relevance of your page.
• Leverage structured citations (NAP, reviews, FAQs) that AI can extract directly for answer snippets.

4. **Monitor AI-Driven Referral Traffic**

Set up analytics to track traffic stemming from AI platforms. Look for referral sources like chat.openai.com or gemini.google.com. This data will reveal whether your content serves as a source and if you’re achieving the desired click-throughs.

5. **Consider Legal & Brand Protections**

If you're concerned about your content being used for training purposes, stay informed about the forthcoming IETF draft and be prepared to adopt any new robots.txt directives once they become standardized. In the meantime, a well-defined terms-of-service page can clarify your position on data usage.

---

Actionable Checklist for Small Business Owners

• ✅ Review and tighten your robots.txt to block known AI crawlers that you want to exclude.
• ✅ Incorporate structured data (schema.org) for every location, product, and service.
• ✅ Create FAQ pages that concisely address common inquiries in a citation-friendly manner.
• ✅ Establish Google Search Console and Bing Webmaster Tools alerts for new crawl anomalies.
• ✅ Monitor AI referral traffic in Google Analytics (create a custom source/medium).
• ✅ Stay updated on the IETF AI-pref draft and be ready to implement new directives.

---

Key Highlights

• Robots.txt only blocks crawling, not AI training.
• To block a specific AI bot, you must list its user-agent token.
• An IETF draft may soon allow sites to signal “no-AI-training” preferences.
• AI tools can still cite your content even if crawlers are blocked.
• Structured data and citation-ready content enhance AI-driven visibility.

---

Why It Matters

The manner in which customers discover local businesses is transforming, shifting from keyword-based searches to conversational AI interactions. When a user queries an AI assistant for recommendations, the responses are generated from extensive language models trained on publicly available web content. If your site's information isn't presented in a way that AI models can easily reference—via schema markup, clear FAQs, and consistent citations—you risk being mentioned in an answer without a clickable link, which can lead to lost traffic and potential leads.

VisibilityAI is committed to helping small businesses gain visibility and citations through these AI agents. By aligning your SEO strategy with AI-friendly best practices—such as maintaining an effective robots.txt, implementing schema markup, and utilizing AI-specific analytics—you ensure that when a user asks, “Where can I find a good bakery in Austin?” the response includes a direct link to your site, rather than just a generic text snippet.

---

FAQs

1. Q: Does blocking Googlebot with robots.txt stop my content from being used by ChatGPT?

A: No. Robots.txt only prevents Googlebot from fetching the page. Other crawlers, including those used by OpenAI or Google’s Gemini, can still access publicly available content unless you specifically block their user-agent.

2. Q: How can I make sure AI tools link back to my site?

A: Publish clear, structured data (schema.org), utilize concise FAQ sections, and ensure your NAP (Name, Address, Phone) information is consistent across the web. AI models are more likely to cite pages that provide clean, authoritative signals.

3. Q: Will the upcoming IETF draft automatically protect my content?

A: Not yet. The draft proposes a standard way to express “no-AI-training” preferences, but until it becomes an official RFC and is adopted by major crawlers, you’ll need to rely on existing robots.txt rules and legal notices.