Google-Extended: Control AI Use of Your Site Without Losing Search

What Happened

Google recently unveiled a new robots.txt directive named Google‑Extended. Unlike the classic User‑agent: Googlebot rules that only tell the crawler whether it may fetch a page, this token reaches Google’s AI‑focused pipelines. It lets you decide if the page can:

1. Train future Gemini models that power Google’s AI apps and the Vertex AI API.

2. Ground Gemini responses with live, cited information drawn from the Google Search index.

Publishers asked for a way to stay visible in search while refusing to feed their content into generative‑AI services, and Google responded with this opt‑out mechanism.

Key Details

• Implementation – Insert a User-agent: Google‑Extended block in your robots.txt. Google applies the most specific rule it finds and refreshes the file from cache at most once every 24 hours.
• Opt‑out flexibility – You can block every path with Disallow: / or target individual directories or pages. For instance, you might let an archived press release be used for training while keeping brand‑new product pages out of AI training.
• Search remains unchanged – Googlebot continues to crawl and index your site unless you explicitly block it with the usual Disallow or Noindex directives. The token influences only Gemini and Vertex AI.
• No extra crawler traffic – Google‑Extended is not a separate crawler; no requests ever hit your server, so your logs stay the same.
• Grounding vs. training – Grounding supplies real‑time citations when an AI answer is generated. Training is a one‑time ingestion that shapes future model releases.
• No ranking signal – Using the token does not affect your position in ordinary Google Search results.

What It Means For Your Business

1. Preserve Your Brand’s Voice in AI

A local restaurant, boutique, or service provider can keep its name appearing in AI‑generated answers by simply leaving the Google‑Extended block permissive. Gemini will then be able to cite your menu, hours, or service descriptions, driving traffic without harming search rankings.

2. Shield Sensitive or Proprietary Information

If you store confidential pricing tables, upcoming product announcements, or internal data, a Disallow rule under the Google‑Extended block keeps those URLs out of AI training pipelines while public pages stay searchable.

3. Tap Into AI‑First Discovery

Customers increasingly start their research with tools like ChatGPT, Perplexity, Gemini, or Google AI Overviews. When your content is eligible for grounding, those tools can quote your site directly, giving you a citation channel that traditional SEO cannot replicate.

4. Manage Crawl Budget Efficiently

Because the token does not generate additional requests, you can fine‑tune which pages participate in AI training without worrying about extra load on your server or a shift in crawl budget.

5. Future‑Proof Your Strategy

Google’s AI ecosystem is still evolving. By mastering Google‑Extended today, you’ll be ready to adapt to upcoming policy shifts around data usage, privacy, and AI‑generated content.

How to Add Google‑Extended to Your Robots.txt

1. Open the robots.txt file at the root of your website.

2. Insert a block such as:

User-agent: Google-Extended

Disallow: /private-content/

3. Save the file and upload it to your server.

4. Use the robots.txt Tester in Google Search Console to confirm the rule is recognized.

Final Thoughts

Google‑Extended offers small and local businesses a precise lever: you can let Gemini cite your pages for AI answers while protecting any sections you consider sensitive. The change requires only a few lines in robots.txt, yet it opens a new discovery path that does not jeopardize your conventional search visibility.

---

Take Action: Review your content inventory, decide which URLs you want AI‑enabled, and update your robots.txt accordingly. Keep an eye on VisibilityAI for further guidance on AI search and citation best practices.