Add rule: Do you let AI answer engines crawl your site? - #13167
Open
isaaclombardssw wants to merge 2 commits into
Open
Add rule: Do you let AI answer engines crawl your site?#13167isaaclombardssw wants to merge 2 commits into
isaaclombardssw wants to merge 2 commits into
Conversation
Contributor
🔍 Preview PR ChangesView updated pages in edit mode:
|
isaaclombardssw
force-pushed
the
isaac/allow-ai-answer-engines
branch
from
August 12, 2026 23:47
16a834b to
2fa52cb
Compare
isaaclombardssw
force-pushed
the
isaac/allow-ai-answer-engines
branch
from
August 13, 2026 02:01
2fa52cb to
78e0990
Compare
github-actions
Bot
added
Age: 🐣 - Young
tiagov8
force-pushed
the
isaac/allow-ai-answer-engines
branch
from
August 13, 2026 18:04
78e0990 to
057a91d
Compare
tiagov8
requested changes
Aug 13, 2026
isaaclombardssw
added a commit
that referenced
this pull request
Aug 14, 2026
> 1. What triggered this change? (PBI link, Email Subject, conversation + reason, etc) ✏️ AEO/GEO rules are multiplying and were scattered between Rules to Better Google Rankings and SEO and Rules to Better AI Development. They now warrant their own category. > 2. What was changed? ✏️ New category `Rules to Better AEO and GEO` under Artificial Intelligence, indexing 10 rules. **Moved out of their old category** (these are purely about AI answer engines): * `ai-optimization-geo-aeo` (was in SEO) * `make-your-website-llm-friendly` (was in AI Development) **Dual-listed**, because they are genuinely both classic SEO and AEO/GEO, and removing them from SEO would leave gaps there: * `page-indexed-by-google` - indexability is the eligibility gate for AI Overviews * `structured-data` - reinforces entities * `author-e-e-a-t` - trust signals * `cite-your-sources`, `back-claims-with-data`, `use-quotations` - the three methods the GEO study measured * `query-deserves-freshness` - recency * `optimise-content-for-topical-authority` - depth Left in SEO only: the technical fundamentals with no AI-specific angle (HTTPS, canonical, titles, meta, headings, responsive, alt text, Open Graph, sitemap, robots.txt). Two follow-ups worth flagging: 1. `do-you-phrase-the-heading-as-a-question` looked like a good fit but is `isArchived: true`, so I left it alone. It may be worth unarchiving, since question-style headings matter more now than when it was retired in 2021. 2. `use-robots-txt-effectively` is from 2015 and only covers redirect folders. Its AI-crawler gap is filled by #13167, but the rule itself could use a refresh. Validators run locally and passing: check-mdx, category-sync, lint-urls. > 3. I paired or mob programmed with: ✏️ n/a 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
isaaclombardssw
force-pushed
the
isaac/allow-ai-answer-engines
branch
from
August 14, 2026 01:40
057a91d to
4f896d8
Compare
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
Age: 🍗 - Ancient
Contributor
|
Howzit @isaaclombardssw, This PR has been here a while. Did you know you should avoid merge debt?
Thanks! |
tiagov8
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✏️ Building the AI Search Readiness tool for tina.io. It checks robots.txt per AI crawler and had no SSW rule to link to.
make-your-website-llm-friendlyonly coversllms.txt, anduse-robots-txt-effectivelywas written in 2015 and only covers redirect folders.✏️ New rule at
/allow-ai-answer-enginescovering:OAI-SearchBot,Claude-User,Claude-SearchBot,PerplexityBot,Bingbot) and training crawlers (GPTBot,ClaudeBot,Google-Extended,CCBot,Applebot-Extended), with what blocking each one actually costsrobots.txtexamples, since the common mistake is one blanket rule that blocks citation along with trainingGoogle-Extendeddoes not remove you from AI Overviews or AI Mode, which is a widespread myth. Those follow ordinary Googlebot and snippet rulessearch,ai-input,ai-train) as the way to state what may be done with content once fetched*, longest path match wins,AllowbeatsDisallowon a tieAdded to Rules to Better Google Rankings and SEO.
Sources cited in the rule: OpenAI bots, Anthropic crawlers, Google crawlers overview, Content Signals Policy, RFC 9309.
Validators run locally and passing: frontmatter, check-mdx, category-sync, lint-urls.
✏️ n/a
🤖 Generated with Claude Code