Content negotiation for the agentic web

Agents are reading your site. What do they see?

AI agents don't render your site. They read it. When they request your pages with Accept: text/markdown, most sites dump raw HTML. Nav bars, cookie banners, and all. We audit what agents actually get back.

Average score across 87 sites audited: 36/100

agent-ready — curl

$ curl -H 'Accept: text/markdown' \

https://example.com/blog

HTTP/1.1 200 OK

Content-Type: text/html; charset=utf-8

Content-Length: 284731

<!DOCTYPE html>

<head>

<title>Our Blog | Example Co.</title>

</head>

<body>

Agent readability score: 12 / 100

Why agents struggle with most websites

AI agents aren't browsers. They don't render CSS or run JavaScript. They read content. Most sites make that needlessly hard.

Agent visits your page

An AI agent requests your URL to extract information or take action.

It receives 94 KB of HTML

HTML

Navigation, cookie banners, ads, scripts, and markup. All noise to the agent.

<!DOCTYPE html><html><head><meta charset="utf-8"><meta name="viewport"...
<script src="/analytics.js"></script><script src="/ads.js"></script>
<nav class="navbar"><ul><li>Home</li><li>About</li><li>Pricing</li>...
<div class="cookie-banner">We use cookies to...</div>
<header class="hero"><div class="container"><h1 class="headline">...

It extracts ~2 KB of signal

Stripped

After stripping noise, only a fraction of the page is actually useful content.

Title: Acme - Project Management Software
Body text: Acme helps teams ship faster...
[navigation links stripped]
[footer stripped]
[12 script tags stripped]

What it should have gotten

Markdown

A markdown-ready site sends clean, structured content from the start. No parsing required.

---
title: Acme - Project Management Software
description: Acme helps teams ship faster with...
---

# Acme

Acme helps teams ship faster with async workflows...

## Features
- Kanban boards, timeline views, and sprints
- GitHub and Slack integrations built-in

AgentReady tests whether your site responds correctly to Accept: text/markdown so agents get clean content, not a parsing job.

Why this matters

You optimize for Google. You optimize for mobile. The next wave of traffic comes from AI agents, and they need clean, structured content, not raw HTML.

Content Negotiation Test

text/markdownthe header agents send

We send Accept: text/markdown, the same header AI agents use. If your server ignores it and returns HTML, agents are forced to parse DOM soup, bloating context and burning tokens.

Context Bloat Detection

~70%of most responses is noise

Nav bars, footers, JavaScript, cookie banners. None of it helps an agent. We measure how much noise your pages carry versus clean, structured markdown an LLM can actually use.

Agent Discovery Audit

Sitemap + Frontmatterhow agents navigate your site

Can agents find your pages? We check for sitemaps, clean link hierarchies, and frontmatter metadata that lets agents understand your content without scraping blind.

How it works

Three steps. Results in under a minute.

Enter your URL

We start from your homepage, discover linked pages, and crawl up to 10, just like an agent exploring your site.

We fetch like an agent

Each page is requested twice: once as HTML, once with Accept: text/markdown. We compare what humans see versus what agents get.

Accept:text/html200

Accept:text/markdown200

See what to fix

Get a per-page score with specific failures like missing content negotiation, context bloat, and no frontmatter. Know exactly what to improve.

/pricing92

/docs/api61

/blog/hello18

Want a quick look first?

Preview what a single page looks like to AI agents. HTML versus clean markdown, side by side. Takes 2 seconds.

What agents receiveHTML

<!DOCTYPE html>
<html lang="en">
<head>
  <title>Acme Inc — Project Management</title>
  <script src="/analytics.js"></script>
  <script src="/hotjar.js"></script>
  <link rel="stylesheet" href="/styles.css">
</head>
<body>
  <nav class="navbar sticky-top">
    <ul><li>Home</li><li>Pricing</li>…</ul>
  </nav>
  <div class="cookie-consent-banner">
    We use cookies to improve your experience…
  </div>
  <main>
    <h1>Ship faster with Acme</h1>
    <p>Acme helps teams coordinate work…</p>
  </main>
  <footer>© 2026 Acme Inc. All rights…</footer>
  <!-- 2,847 more lines… -->
</body>

What they should getMarkdown

---
title: "Acme Inc — Project Management"
description: "Acme helps teams coordinate work"
---
 
# Ship faster with Acme
 
Acme helps teams coordinate work with
async-first collaboration tools.
 
## Features
 
- Kanban boards and timeline views
- GitHub and Slack integrations
- Real-time multiplayer editing
 
## Getting Started
 
Sign up at [acme.dev](https://acme.dev)
and create your first project in seconds.

—signal in HTML

—signal in Markdown

Preview a page

From the blog

Learn more about the agentic web and how to prepare your site.

Apr 13, 2026·4 min read