Do AI crawlers like gptbot support content negotiation for markdown?
No. As of 2026-07, OpenAI's official crawler documentation says nothing about Accept headers, content negotiation, or markdown — GPTBot, OAI-SearchBot, and ChatGPT-User are documented only by user agent, purpose, and IP range. Don't build content negotiation for them. Serve good HTML, and optionally publish parallel .md URLs plus llms.txt.
Why — the first-principles explanation
Content negotiation is an old, correct idea from HTTP: the client sends `Accept: text/markdown`, the server checks it, and returns the best matching representation of the same resource. It works beautifully in theory. In practice it's one of the least-used parts of HTTP, because it requires both sides to implement it and neither side benefits until the other does. That chicken-and-egg problem has kept it niche for thirty years, and AI crawlers haven't broken the deadlock.
Why would crawlers even want markdown? Because HTML is mostly noise to them. A modern page is 200KB of nav bars, cookie banners, scripts, and tracking wrapped around 3KB of actual prose. Every crawler runs a boilerplate-stripping pipeline to throw all that away and recover the text. Markdown is roughly what falls out at the end of that pipeline. So the appealing idea is: skip the demolition, just hand them the text. Sensible. It saves both sides bandwidth and CPU.
But here's why it hasn't happened: the crawler operator has no incentive to trust you. If a server can return different content based on an Accept header, it can return different content — nicer, keyword-stuffed, or simply not what humans see. That's cloaking, and search engines have fought it for twenty-five years. A crawler that requests markdown is asking to be shown a version no human ever verified. The safest engineering choice is to fetch what a browser fetches, and eat the boilerplate-stripping cost. The reason content negotiation doesn't get adopted here isn't technical difficulty — it's that fetching-what-humans-see is a trust property worth more than the bandwidth savings.
What's actually emerging instead is the parallel URL convention: publish `/article` as HTML and `/article.md` as markdown, discoverable via a link or an `llms.txt` index. This dodges the trust problem because the markdown lives at its own address that anyone — human, crawler, or auditor — can fetch and compare. It's a convention, not a standard: no major AI crawler is documented as requiring it, and there's no confirmed ranking or retrieval benefit as of 2026-07. Do it if it's cheap. Don't rebuild your stack for it.
An example that makes it click
Imagine a restaurant that hands a different menu to food critics than to regular diners. Even if the critic's menu were genuinely more convenient — bigger print, better organized — the critic should refuse it. The whole job is to eat what the customers eat.
That's why crawlers ask for the same HTML your browser gets. Not because markdown would be worse. Because a special version handed only to them is, by definition, unverified. If you want them to have the clean text, put it on its own table where everyone can sit — that's the .md URL.
How to do it
- Don't implement Accept-header content negotiation for AI crawlers. OpenAI's official bot documentation makes no mention of Accept headers, content types, or markdown support as of 2026-07 — you'd be building against nothing.
- Serve clean, semantic HTML as the primary representation. This is what every documented crawler actually fetches.
- Make sure content exists without JavaScript. Server-side render or pre-render — crawler JS execution is inconsistent and costly.
- Optional and cheap: publish parallel markdown at a distinct URL (e.g. /post/slug.md) and link to it with <link rel="alternate" type="text/markdown">.
- Optional: add /llms.txt as an index of your key pages and their markdown equivalents. It's a community convention, not a ratified standard, and no crawler is documented as requiring it.
- Control access via robots.txt using the documented user agents — GPTBot for training, OAI-SearchBot for ChatGPT search visibility. Verify real crawler traffic against OpenAI's published IP range files rather than trusting the user-agent string.
- If you do serve markdown, keep it identical in substance to the HTML. Divergence is cloaking, and it's the risk the whole design is trying to avoid.
Key facts
- OpenAI documents three crawlers as of 2026-07: GPTBot (model training, UA contains 'GPTBot/1.4'), OAI-SearchBot (ChatGPT search results, UA contains 'OAI-SearchBot/1.4'), and ChatGPT-User (user-initiated fetches, UA contains 'ChatGPT-User/1.0').
- OpenAI's official crawler documentation provides no specification for Accept headers, content negotiation, content types, or markdown support.
- Each crawler publishes a JSON IP range file for verification: openai.com/gptbot.json, openai.com/searchbot.json, and openai.com/chatgpt-user.json.
- Disallowing GPTBot in robots.txt signals content should not be used for model training; blocking OAI-SearchBot removes a site from ChatGPT search results.
- ChatGPT-User is triggered by user requests rather than automated crawling, so robots.txt may not apply to it — a meaningful distinction when planning access control.
- Serving crawler-specific content that differs from what humans see is cloaking, the long-standing reason crawlers prefer fetching the same representation a browser receives.
▶ The 60-second explainer (script)
Do AI crawlers like GPTBot support content negotiation for markdown? No. As of mid-2026, OpenAI's official crawler docs say nothing about Accept headers, content types, or markdown. They document three bots — GPTBot for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for user-triggered fetches — by user agent, purpose, and IP range. That's it. So don't build content negotiation for them. But the interesting question is why not, because the idea is genuinely good. HTML is mostly noise to a crawler. A modern page is two hundred kilobytes of nav bars, cookie banners, and scripts wrapped around three kilobytes of actual prose. Every crawler runs a stripping pipeline to throw that away. Markdown is basically what falls out the end. So why not skip the demolition and just hand them the text? Here's why. If your server can return different content based on a header, it can return different content. Nicer. Keyword-stuffed. Not what humans see. That's cloaking, and search engines have fought it for twenty-five years. A crawler asking for markdown is asking to be shown a version no human ever verified. So the safe engineering choice is: fetch what a browser fetches, and eat the cost. It's not a technical limitation. It's a trust property that's worth more than the bandwidth. What's emerging instead is parallel URLs — publish slash-article and slash-article-dot-md, index it in llms.txt. That works because the markdown has its own address anyone can check. It's a convention, not a standard, and there's no confirmed benefit. Cheap to do. Don't rebuild your stack for it.
What authoritative sources say
People also ask
Does GPTBot send an Accept header preferring markdown?
OpenAI's documentation doesn't specify any Accept-header behavior for GPTBot. There's no documented markdown preference, so there's nothing to negotiate against.
Should I serve markdown to AI crawlers at all?
Only via a parallel URL like /page.md, never by varying the response to the same URL based on who's asking. Same-URL variance is cloaking, and it's the exact pattern crawlers are designed to distrust.
Does llms.txt actually work?
It's a community convention, not a ratified standard. No major AI crawler is documented as requiring or rewarding it as of 2026-07. It costs almost nothing to publish, but don't expect a measurable effect.
How do I verify a request is really from GPTBot?
Check the source IP against OpenAI's published range file at openai.com/gptbot.json. User-agent strings are trivially spoofed; IP ranges are the actual verification mechanism.
What's the difference between GPTBot and OAI-SearchBot?
GPTBot collects data for training foundation models. OAI-SearchBot powers ChatGPT's search features. Blocking GPTBot keeps you out of training; blocking OAI-SearchBot keeps you out of ChatGPT search results — usually not what publishers want.