diff --git a/categories/artificial-intelligence/rules-to-better-aeo-and-geo.mdx b/categories/artificial-intelligence/rules-to-better-aeo-and-geo.mdx index fc56cc13ada..fc12f53c1bb 100644 --- a/categories/artificial-intelligence/rules-to-better-aeo-and-geo.mdx +++ b/categories/artificial-intelligence/rules-to-better-aeo-and-geo.mdx @@ -9,6 +9,7 @@ index: - rule: public/uploads/rules/page-indexed-by-google/rule.mdx - rule: public/uploads/rules/make-your-website-llm-friendly/rule.mdx - rule: public/uploads/rules/allow-ai-answer-engines/rule.mdx + - rule: public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx - rule: public/uploads/rules/cite-your-sources/rule.mdx - rule: public/uploads/rules/back-claims-with-data/rule.mdx - rule: public/uploads/rules/use-quotations/rule.mdx diff --git a/public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx b/public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx new file mode 100644 index 00000000000..39a3b90baa3 --- /dev/null +++ b/public/uploads/rules/serve-markdown-to-ai-agents/rule.mdx @@ -0,0 +1,101 @@ +--- +type: rule +title: Do you serve Markdown to AI agents? +uri: serve-markdown-to-ai-agents +categories: + - category: categories/artificial-intelligence/rules-to-better-aeo-and-geo.mdx +authors: + - title: Isaac Lombard + url: 'https://www.ssw.com.au/people/isaac-lombard' +related: + - rule: public/uploads/rules/make-your-website-llm-friendly/rule.mdx + - rule: public/uploads/rules/ai-optimization-geo-aeo/rule.mdx + - rule: public/uploads/rules/allow-ai-answer-engines/rule.mdx +guid: e6cca9fd-7129-40a1-b439-b140292b52c8 +seoDescription: AI agents waste context reading HTML wrappers. Learn how to use content negotiation to serve Markdown instead, cutting token cost by up to 80%. +created: 2026-08-12T07:54:54.000Z +--- + +When an AI agent reads a web page, it pays for every byte of markup it does not need. Navigation, scripts, styles and wrapper divs all consume context that could have held your actual content. + +Worse, when a page is too large the agent truncates it. Your answer may be sitting in the half that never got read. + + + +## Use content negotiation + +[Content negotiation](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-negotiation) is a standard part of HTTP. The client states what format it wants, and the server responds in that format. + +When a request carries `Accept: text/markdown`, return Markdown instead of HTML. + + + ```http + GET /blog/my-article HTTP/1.1 + Host: www.example.com + Accept: text/markdown + ``` + + ```http + HTTP/1.1 200 OK + Content-Type: text/markdown; charset=utf-8 + Vary: Accept + + # My article + + The content, with no navigation, scripts or wrappers. + ``` + } + figurePrefix="good" + figure="Good example - The agent asks for Markdown and gets Markdown" +/> + +Cloudflare measured [up to 80% fewer tokens](https://developers.cloudflare.com/changelog/2026-02-12-markdown-for-agents) compared to the HTML equivalent. Cheaper to read, faster to process, and far more likely to be consumed whole rather than truncated. + +## Always send `Vary: Accept` + +This is the trap that bites people. Without `Vary: Accept`, a CDN or browser cache can store the Markdown response and serve it to the next visitor, who gets a wall of plain text instead of your website. + + + If one URL can return more than one format, `Vary: Accept` is mandatory. Skipping it is a caching bug waiting to happen. + } + figurePrefix="none" + figure="" +/> + +## Add a .md URL as a fallback + +Not every agent sends the header. Some coding agents do, most other clients do not. + +So expose the same content at a predictable URL as well. The convention is to append `.md` to the page path. + +| Page | Markdown | +| --- | --- | +| `/blog/my-article` | `/blog/my-article.md` | +| `/` | `/index.md` | + +Do both. The header is the correct mechanism, and the URL is the fallback for agents that have not caught up. + +## This is nearly free on a Markdown-backed site + +If your content is already authored in Markdown, and it is on any Git-based CMS, then the Markdown already exists. You are converting it to HTML at build time and throwing the original away. + +Instead of generating Markdown from your HTML, serve the source you started with. This is a build step or an edge function, not a rewrite. + +## How is this different to llms.txt? + +They solve different problems and you want both. + +* `llms.txt` is **one index file** at the root, listing what your site contains and where to find it. It helps an agent decide what to read +* Markdown negotiation applies to **every page**. It makes the thing the agent decided to read cheap to consume + +## Check your own site + +```bash +curl -sI -H "Accept: text/markdown" https://www.example.com/ | grep -i "content-type\|vary" +``` + +If `Content-Type` comes back as `text/html`, you are not negotiating. Adoption is still low, as Cloudflare's [Agent Readiness score](https://blog.cloudflare.com/agent-readiness/) shows, so this is an easy way to stand out.