Google’s New 503, Robots.txt & Bracket URL Rules: What Small Businesses Need to Know

What Happened

Google Search Central posted a series of updates on October 21, 2026, outlining three key changes that affect how small and local businesses manage their online presence:

1. Use a 503 status code during PHP patch downtime – Instead of a generic 404 or a misleading 200, Google recommends a 503 to preserve indexed URLs.

2. Robots.txt limits for AI crawlers – The new guidelines clarify which AI agents can crawl your site and how to structure robots.txt for optimal AI visibility.

3. Bracket URLs & JPEG‑XL support – Google now treats URLs that contain brackets more predictably and has added support for the next‑generation JPEG‑XL image format.

These changes are part of Google’s ongoing effort to improve AI search results and keep local businesses discoverable across emerging AI platforms like ChatGPT, Gemini, and Google AI.

Key Details

1. 503 Status Code for PHP Patches

• Why 503? A 503 tells Google that your site is temporarily unavailable but still indexed. It prevents Google from dropping URLs from its index.
• When to use it? Serve a 503 for no longer than 1–2 days. After that, Google may treat it like a permanent outage.
• Implementation tips:

- Add a simple PHP file that returns header('HTTP/1.1 503 Service Unavailable'); and a custom message.

- Keep the page lightweight; Google’s crawler will fetch it quickly.

- Avoid generic 404 pages that could confuse both users and crawlers.

2. Robots.txt for AI Crawlers

• New agents: Google’s AI agents, Bing’s AI, and others now read robots.txt just like traditional crawlers.
• Allowed vs. Disallowed:

- Allow: User-agent: * and Allow: / for AI agents that you want indexed.

- Disallow: User-agent: Google-Assistant or User-agent: Bing-AI if you prefer to keep AI content private.

• Crawl‑rate considerations: If you’re serving a 503, ensure that robots.txt doesn’t block AI crawlers from reaching the 503 page.
• Testing: Use Google Search Console’s “Robots.txt Tester” to confirm that AI agents can access the pages you want.

3. Bracket URLs & JPEG‑XL

• Bracket URLs: URLs containing [ ] are now automatically canonicalized by Google. For example, https://example.com/product[123] will be treated as https://example.com/product in search results. This prevents duplicate content issues.
• JPEG‑XL: Google now supports the JPEG‑XL format, which offers smaller file sizes with high quality. If you’re hosting images for product listings, consider converting to JPEG‑XL to reduce bandwidth and improve page load times—an important ranking factor for both web and AI search.

What It Means For Your Business

1. Preserve Local Listings During Updates

When you patch your PHP core (common with WordPress sites), a 503 ensures that Google’s index stays intact. Your local business page won’t drop from search results, keeping your address, hours, and reviews visible to AI assistants.

2. Control AI Visibility

If you want your business to appear in AI‑powered answers, make sure your robots.txt allows the relevant agents. Conversely, if you’re experimenting with private data or want to test AI responses, you can disallow specific agents without affecting normal web crawlers.

3. Reduce Duplicate Content Risks

Bracket URLs can inadvertently create duplicate content if different URLs point to the same page. By letting Google canonicalize them automatically, you avoid penalties that could hurt both web and AI rankings.

4. Faster Page Loads = Better AI Answers

AI assistants often pull data from your site’s structured data and images. Using JPEG‑XL reduces load times, which means faster data retrieval for AI models and a smoother user experience.

5. Stay Ahead of AI Search Trends

VisibilityAI’s Riva tool can track how your brand appears in AI answers. By aligning your site with Google’s new guidelines, you’ll see higher visibility scores and stronger citations across ChatGPT, Gemini, and other AI platforms.

Quick Checklist for Business Owners

• [ ] Serve a 503 during PHP or other critical updates.
• [ ] Verify that your robots.txt permits AI agents you want indexed.
• [ ] Canonicalize bracket URLs or let Google handle them.
• [ ] Convert high‑resolution product images to JPEG‑XL.
• [ ] Use VisibilityAI’s Riva to monitor AI search visibility.

Implementing these steps will help your local business stay discoverable, maintain accurate citations, and thrive in the evolving AI search landscape.

Webinar: AI Shopping 101

Don’t miss our upcoming webinar on AI Shopping: Get Your Products Found in Google & ChatGPT. Save your spot today to learn how AI agents decide which products to recommend and how to optimize your listings for maximum impact.

---

For more detailed guidance, read the full SEO Pulse article on Search Engine Journal.

Why It Matters:

When local businesses rely on AI assistants to answer customer questions, every detail of how your site is crawled and indexed matters. A 503 during maintenance preserves your current rankings, ensuring that customers can still find your address, hours, and reviews in AI responses. Meanwhile, properly configured robots.txt guarantees that AI agents can access the content you want them to recommend, giving you full control over your brand’s voice in AI answers.

With AI search growing faster than traditional web search, businesses that adapt quickly gain a competitive edge. By embracing JPEG‑XL images and managing bracket URLs, you reduce page load times and avoid duplicate content penalties—both critical for ranking well in AI‑driven result panels. VisibilityAI’s Riva tool can then track these changes, showing you exactly how your brand’s AI visibility improves over time.