From d94d581c53d071c1e46c379ffe3ba369205b4f02 Mon Sep 17 00:00:00 2001 From: isaaclombardssw Date: Thu, 13 Aug 2026 09:24:25 +1000 Subject: [PATCH 1/2] Add rule: Do you serve Markdown to AI agents? Co-Authored-By: Claude Opus 5 (1M context) --- .../rules-to-better-aeo-and-geo.mdx | 1 + .../serve-markdown-to-ai-agents/rule.mdx | 112 ++++++++++++++++++ 2 files changed, 113 insertions(+) create mode 100644 public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx diff --git a/categories/artificial-intelligence/rules-to-better-aeo-and-geo.mdx b/categories/artificial-intelligence/rules-to-better-aeo-and-geo.mdx index fc56cc13ada..fc12f53c1bb 100644 --- a/categories/artificial-intelligence/rules-to-better-aeo-and-geo.mdx +++ b/categories/artificial-intelligence/rules-to-better-aeo-and-geo.mdx @@ -9,6 +9,7 @@ index: - rule: public/uploads/rules/page-indexed-by-google/rule.mdx - rule: public/uploads/rules/make-your-website-llm-friendly/rule.mdx - rule: public/uploads/rules/allow-ai-answer-engines/rule.mdx + - rule: public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx - rule: public/uploads/rules/cite-your-sources/rule.mdx - rule: public/uploads/rules/back-claims-with-data/rule.mdx - rule: public/uploads/rules/use-quotations/rule.mdx diff --git a/public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx b/public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx new file mode 100644 index 00000000000..6e9b913d150 --- /dev/null +++ b/public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx @@ -0,0 +1,112 @@ +--- +type: rule +title: Do you serve Markdown to AI agents? +uri: serve-markdown-to-ai-agents +categories: + - category: categories/artificial-intelligence/rules-to-better-aeo-and-geo.mdx +authors: + - title: Isaac Lombard + url: 'https://www.ssw.com.au/people/isaac-lombard' +related: + - rule: public/uploads/rules/make-your-website-llm-friendly/rule.mdx + - rule: public/uploads/rules/ai-optimization-geo-aeo/rule.mdx +guid: e6cca9fd-7129-40a1-b439-b140292b52c8 +seoDescription: AI agents waste context reading HTML wrappers. Learn how to use content negotiation to serve Markdown instead, cutting token cost by up to 80%. +created: 2026-08-12T07:54:54.000Z +--- + +When an AI agent reads a web page, it pays for every byte of markup it does not need. Navigation, scripts, styles and wrapper divs all consume context that could have held your actual content. + +Worse, when a page is too large the agent truncates it. Your answer may be sitting in the half that never got read. + + + +## Use content negotiation + +Content negotiation is a standard part of HTTP. The client states what format it wants, and the server responds in that format. + +When a request carries `Accept: text/markdown`, return Markdown instead of HTML. + + + ```http + GET /blog/my-article HTTP/1.1 + Host: www.example.com + Accept: text/markdown + ``` + + ```http + HTTP/1.1 200 OK + Content-Type: text/markdown; charset=utf-8 + Vary: Accept + + # My article + + The content, with no navigation, scripts or wrappers. + ``` + } + figurePrefix="good" + figure="Good example - The agent asks for Markdown and gets Markdown" +/> + +Cloudflare measured up to 80% fewer tokens compared to the HTML equivalent. Cheaper to read, faster to process, and far more likely to be consumed whole rather than truncated. + +## Always send Vary: Accept + +This is the trap that bites people. Without `Vary: Accept`, a CDN or browser cache can store the Markdown response and serve it to the next visitor, who gets a wall of plain text instead of your website. + + + If one URL can return more than one format, `Vary: Accept` is mandatory. Skipping it is a caching bug waiting to happen. + } + figurePrefix="none" + figure="" +/> + +## Add a .md URL as a fallback + +Not every agent sends the header. Some coding agents do, most other clients do not. + +So expose the same content at a predictable URL as well. The convention is to append `.md` to the page path. + +| Page | Markdown | +| --- | --- | +| `/blog/my-article` | `/blog/my-article.md` | +| `/` | `/index.md` | + +Do both. The header is the correct mechanism, and the URL is the fallback for agents that have not caught up. + +## This is nearly free on a Markdown-backed site + +If your content is already authored in Markdown, and it is on any Git-based CMS, then the Markdown already exists. You are converting it to HTML at build time and throwing the original away. + +Instead of generating Markdown from your HTML, serve the source you started with. This is a build step or an edge function, not a rewrite. + +## How is this different to llms.txt? + +They solve different problems and you want both. + +* `llms.txt` is **one index file** at the root, listing what your site contains and where to find it. It helps an agent decide what to read +* Markdown negotiation applies to **every page**. It makes the thing the agent decided to read cheap to consume + +## Check your own site + +```bash +curl -sI -H "Accept: text/markdown" https://www.example.com/ | grep -i "content-type\|vary" +``` + +If `Content-Type` comes back as `text/html`, you are not negotiating. Adoption is still low, so this is an easy way to stand out. + +## Related rules + +* [Do you provide an llms.txt file to make your website LLM-friendly?](/make-your-website-llm-friendly) +* [Do you optimize your content for AI answers?](/ai-optimization-geo-aeo) +* [Do you let AI answer engines crawl your site?](/allow-ai-answer-engines) + +## References + +* [Cloudflare, Markdown for agents](https://developers.cloudflare.com/changelog/2026-02-12-markdown-for-agents) +* [Cloudflare, Agent Readiness score](https://blog.cloudflare.com/agent-readiness/) +* [RFC 9110, content negotiation](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-negotiation) From bf5cd3331c565520ad33fe34cf297258f74f99d0 Mon Sep 17 00:00:00 2001 From: isaaclombardssw Date: Fri, 14 Aug 2026 11:45:57 +1000 Subject: [PATCH 2/2] Move related rules to frontmatter, inline sources, code-format heading Co-Authored-By: Claude Opus 5 (1M context) --- .../serve-markdown-to-ai-agents/rule.mdx | 21 +++++-------------- 1 file changed, 5 insertions(+), 16 deletions(-) diff --git a/public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx b/public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx index 6e9b913d150..39a3b90baa3 100644 --- a/public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx +++ b/public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx @@ -10,6 +10,7 @@ authors: related: - rule: public/uploads/rules/make-your-website-llm-friendly/rule.mdx - rule: public/uploads/rules/ai-optimization-geo-aeo/rule.mdx + - rule: public/uploads/rules/allow-ai-answer-engines/rule.mdx guid: e6cca9fd-7129-40a1-b439-b140292b52c8 seoDescription: AI agents waste context reading HTML wrappers. Learn how to use content negotiation to serve Markdown instead, cutting token cost by up to 80%. created: 2026-08-12T07:54:54.000Z @@ -23,7 +24,7 @@ Worse, when a page is too large the agent truncates it. Your answer may be sitti ## Use content negotiation -Content negotiation is a standard part of HTTP. The client states what format it wants, and the server responds in that format. +[Content negotiation](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-negotiation) is a standard part of HTTP. The client states what format it wants, and the server responds in that format. When a request carries `Accept: text/markdown`, return Markdown instead of HTML. @@ -50,9 +51,9 @@ When a request carries `Accept: text/markdown`, return Markdown instead of HTML. figure="Good example - The agent asks for Markdown and gets Markdown" /> -Cloudflare measured up to 80% fewer tokens compared to the HTML equivalent. Cheaper to read, faster to process, and far more likely to be consumed whole rather than truncated. +Cloudflare measured [up to 80% fewer tokens](https://developers.cloudflare.com/changelog/2026-02-12-markdown-for-agents) compared to the HTML equivalent. Cheaper to read, faster to process, and far more likely to be consumed whole rather than truncated. -## Always send Vary: Accept +## Always send `Vary: Accept` This is the trap that bites people. Without `Vary: Accept`, a CDN or browser cache can store the Markdown response and serve it to the next visitor, who gets a wall of plain text instead of your website. @@ -97,16 +98,4 @@ They solve different problems and you want both. curl -sI -H "Accept: text/markdown" https://www.example.com/ | grep -i "content-type\|vary" ``` -If `Content-Type` comes back as `text/html`, you are not negotiating. Adoption is still low, so this is an easy way to stand out. - -## Related rules - -* [Do you provide an llms.txt file to make your website LLM-friendly?](/make-your-website-llm-friendly) -* [Do you optimize your content for AI answers?](/ai-optimization-geo-aeo) -* [Do you let AI answer engines crawl your site?](/allow-ai-answer-engines) - -## References - -* [Cloudflare, Markdown for agents](https://developers.cloudflare.com/changelog/2026-02-12-markdown-for-agents) -* [Cloudflare, Agent Readiness score](https://blog.cloudflare.com/agent-readiness/) -* [RFC 9110, content negotiation](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-negotiation) +If `Content-Type` comes back as `text/html`, you are not negotiating. Adoption is still low, as Cloudflare's [Agent Readiness score](https://blog.cloudflare.com/agent-readiness/) shows, so this is an easy way to stand out.