You added llms.txt. Your pages are still 500KB.
Robots rules, llms.txt, structured data, and content negotiation solve four different problems. Here is what each signal does—and what to build first.
Someone asks whether your website is ready for AI. You add an llms.txt file, ship it, and move on.
There is just one problem: every URL in that beautiful new index still returns the same enormous HTML document.
llms.txt is useful. So are robots.txt, structured data, and content negotiation. They are also routinely discussed as if they are rival answers to one question. They are not. They solve four different problems.
The four signals
Start with the map:
| Signal | Solves | Scope |
|---|---|---|
robots.txt | Access control | Which pages can be fetched at all |
llms.txt | Site overview | What your product is and where to find things |
| Structured data | Semantic metadata | What individual pages mean |
| Content negotiation | Format efficiency | What agents receive when they fetch a page |
The distinctions matter because a site can implement one perfectly and remain terrible at the other three.
robots.txt: access control
robots.txt has been around since 1994. It tells crawlers which parts of your site they're allowed to fetch.
User-agent: *
Disallow: /admin/
Disallow: /api/internal/
User-agent: GPTBot
Disallow: /
This is a blunt instrument by design. It is about access, not representation. You can disallow a crawler or a path. You cannot say, “You may fetch this, but please take the lightweight version.”
One important caveat: robots.txt communicates a preference to compliant crawlers; it is not authentication or an enforcement boundary. Use real access controls for private content.
llms.txt: the site overview
llms.txt is a proposed convention, not an official web standard. The idea is pleasantly simple: publish a Markdown file at /llms.txt that introduces your site and links to the important parts.
# Acme Docs
> Developer documentation for Acme's API platform.
## Docs
- [Getting Started](/docs/getting-started): Authentication and first API call
- [API Reference](/docs/api): Full endpoint reference
- [SDKs](/docs/sdks): Client libraries for Python, TypeScript, Go
## Support
- [Status](https://status.acme.com): Current API status
- [GitHub](https://github.com/acme/acme): Open source SDKs
Think of it as a README for your website. It answers two questions: “What is this place?” and “Where should I look next?”
That is genuinely useful. It is especially useful for documentation and product sites where the navigation hierarchy carries meaning.
Structured data: semantic metadata
Structured data (JSON-LD, typically using Schema.org) embeds machine-readable metadata directly in your HTML. It's the reason Google shows star ratings in search results.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "How to deploy with zero downtime",
"datePublished": "2026-03-15",
"author": { "@type": "Person", "name": "Jane Smith" }
}
</script>
This gives machines explicit facts about the page: its type, headline, author, publish date, product, price, or whatever else the schema describes. Search engines have used it for years; agent systems can use it too.
Structured data is worth having. But like llms.txt, it's metadata layered on top of HTML. It doesn't change what agents receive. They still get the whole document.
Content negotiation: format efficiency
This is the layer we think most sites underinvest in.
HTTP has supported content negotiation since the early days. The Accept header lets a client declare what format it wants. Most browsers send Accept: text/html. Claude Code, Perplexity, and many modern agent frameworks send:
Accept: text/markdown, text/html, */*
If your server respects that preference, it can return clean Markdown instead of the full browser document. Same URL. Same underlying content. Far less machinery.
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
---
title: How to deploy with zero downtime
date: 2026-03-15
author: Jane Smith
---
# How to deploy with zero downtime
Your actual content here...
We unpacked the payload difference in our introduction to content negotiation. Your site will produce its own numbers. The useful question is how much browser-only material you can avoid sending at all.
What to implement first
For a public content or documentation site, we would usually work in this order:
1. Content negotiation: Improve the thing every page request returns. Check the Accept header, render the content without browser chrome, add useful frontmatter, and send the correct Content-Type. The idea is simple; production caching and framework details deserve care.
2. llms.txt: Once the destinations are readable, publish a good map. Keep it selective. A list of 4,000 unlabeled links is a sitemap wearing a Markdown costume.
3. Structured data: Add explicit semantics for content types that map cleanly to Schema.org, including articles and products. This is often low effort when the page already comes from structured CMS fields.
4. robots.txt: You probably have one. Review it intentionally. Training crawlers, search crawlers, and user-directed agents are not always the same thing, and blanket rules can block traffic you meant to allow.
That ordering is not universal. If your immediate goal is controlling crawler access, start with robots.txt. If agent discovery is the only missing piece on an otherwise excellent docs site, start with llms.txt. Prioritize the problem you actually have.
Where sites actually fail
The same failure modes keep showing up in audits:
No content negotiation at all: The most common case. Every request returns the same HTML regardless of Accept. Agents are forced to parse DOM soup.
Content type set, body unchanged: A server returns Content-Type: text/markdown but the body is still raw HTML. Often a middleware misconfiguration.
Navigation not stripped: The server returns Markdown, but it's the entire HTML document converted verbatim. Nav bars, headers, footers, scripts. The size delta is minimal; agents get all the noise.
llms.txt exists, content negotiation doesn't: A site has a polished llms.txt and zero support for the Accept header. The overview is good; every page visit is still inefficient.
The last one is the sneakiest. llms.txt is visible, concrete, and easy to ship. Content negotiation touches the rendering layer, so it gets deferred. The site gains a map, but none of the roads improve.
The complete picture
If you want to think about agent readiness as a stack:
robots.txt → Can agents access your site?
llms.txt → Can agents understand what your site contains?
Structured data → Can agents understand what individual pages mean?
Content negotiation → Can agents read your pages efficiently?
All four layers can matter. Per-page format efficiency is where we see the largest, most measurable gap: what did the client request, what did the server return, and how much of that response was useful?
See where you stand
AgentReady.dev focuses on that content-negotiation layer. It requests up to 10 pages in both formats and shows whether agents receive clean Markdown or the same HTML with a more optimistic header.
It is free. No signup. About 30 seconds. Your llms.txt file is welcome to watch.