NewIntroducing Agent Layer: serve agent ready content and see how agents use itRead the post
← All posts
·8 min read

Inside the audit: how we score a website for AI agents

We fetch every page twice, run seven checks, and turn the results into one readiness score. Here is the full pipeline.

engineeringagentscrawlerscoring

What happens after you paste a URL into AgentReady.dev?

We crawl the site, ask for each page in two formats, and look for the difference between “technically returned something” and “actually useful to an agent.” Then seven checks become one score.

Here is the full pipeline—including the judgment calls hiding behind that tidy number.

The audit pipeline

Every audit runs through four stages: crawl, fetch both formats, score each page, and aggregate.

We start with your root URL and discover links. Then we fetch every page twice — once as a browser would, once as an AI agent would. We score each pair of responses against seven checks. Finally, we compute a single site score, weighted by page importance.

Here's the skeleton of that workflow:

async function runAudit(jobId: string) {
  // 1. Fetch root page, discover same-domain links
  // 2. For each page: fetch HTML + Markdown
  // 3. Score each page
  // 4. Aggregate: home page counts 2x
}

Four steps. Plenty of places for a website to surprise us.

Step 1: Crawling

We fetch your root URL with a standard Accept: text/html header and parse the response with Cheerio to extract all same-domain links.

A few crawler decisions matter more than they might seem:

Locale-prefixed paths get deprioritized. Many sites serve the same content under /en-gb/, /de-de/, and so on. We detect locale patterns and push those links to the back of the queue, so the canonical English paths get audited first. If a site only has locale-prefixed URLs, we'll still crawl them — they're not excluded, just ranked lower.

We stop at 10 pages. This is a readiness audit, not a complete inventory. A homepage and a representative set of internal pages usually reveal whether content negotiation is a system or a one-off. Crawling another 90 near-identical templates rarely changes the diagnosis.

We enforce hard safety limits per request. Each fetch has a 15-second timeout and a 5MB response size cap. We also validate URLs against SSRF protection before fetching — no private IPs, no internal hostnames.

Once we have the link list, the crawler moves to the scoring phase.

Step 2: Fetching both formats

This is the entire premise of the audit. Every page gets two requests:

GET /page
Accept: text/html

GET /page
Accept: text/markdown, text/html, */*

The first asks for the browser representation. The second prefers Markdown while allowing HTML as a fallback. If your server sends the same HTML document both times, we have learned something useful before scoring anything else.

We intentionally send a realistic preference list rather than requesting Markdown alone. A production client often needs a fallback. A server that only works for the neatest possible test has not finished the job.

Step 3: Scoring

Every page runs through seven checks. Most resolve to pass, partial, or fail; link quality uses a proportional score. We weight the results and calculate a page score from 0–100.

CheckWeightWhat it tests
Markdown Response20Did the server respond differently to the Markdown request?
Valid Markdown20Is the Markdown response actually Markdown, not HTML?
Navigation Stripped15Were nav bars, headers, and footers removed?
YAML Frontmatter15Does the response include structured metadata?
Sitemap / Index10(Home only) Does the root page act as a link directory?
Link Quality10Are the links in the Markdown well-formed?
Size Delta10Is the Markdown meaningfully smaller than the HTML?

Let's go through each one.

Markdown Response (weight: 20)

The table-stakes check. We compare the HTML and Markdown requests to see whether the server negotiated a different representation. If the supposed Markdown request still returns text/html, the site has not implemented content negotiation.

This check is binary: pass or fail.

Valid Markdown (weight: 20)

A server can claim Content-Type: text/markdown and still send raw HTML. Yes, really. Headers are promises, not proof, so we inspect the body.

We scan for positive Markdown signals: headings (#), links ([text](url)), lists, code blocks, blockquotes, bold text. Then we check for disqualifying HTML signals: <!DOCTYPE>, <html>, <head>, <body>. If the response looks like HTML, it fails — regardless of what the header says.

Navigation Stripped (weight: 15)

Markdown is not automatically clean. If the response still contains navigation, headers, footers, scripts, and styles, the server may have converted the whole document without deciding what belongs in the content. That preserves most of the noise we were trying to remove.

We check for <nav>, <header>, <footer>, nav-class and nav-id attributes, <script>, and <style> tags. Any of these in the response body triggers a fail.

YAML Frontmatter (weight: 15)

Clean Markdown is useful. Clean Markdown that identifies itself is better. Frontmatter gives agents the title, description, date, author, or canonical URL without asking them to infer those details from the prose.

We check that the response starts with ---, and that the frontmatter block contains at least two meaningful fields from: title, description, date, author, tags, url, canonical. A response that opens with --- but only has one field gets a partial.

Sitemap / Index (weight: 10, home page only)

This check only runs on the root URL. We ask one question: can an agent use this response to figure out where to go next?

We're looking for a link directory — five or more organized markdown links in list format pointing to major sections of the site. When an agent starts exploring a new site, it often reads the root page to decide where to go next. A standard HTML homepage gives it cards and hero images it can't use. A structured index gives it a roadmap.

Link Quality (weight: 10)

Links are how an agent keeps exploring. We flag three patterns that turn them into dead ends:

  • Empty link text — [](url) tells an agent nothing about the destination
  • Generic text — "click here", "read more", "link" are meaningless out of context
  • Non-functional hrefs — #, javascript:, and empty hrefs can't be followed

This check produces a continuous score from 0.0 to 1.0 based on the proportion of problematic links. A page with no link issues scores 1.0. A page where half the links are generic anchor text scores 0.5.

Size Delta (weight: 10)

If your Markdown response is almost the same size as the HTML, we get suspicious. The whole point is to remove the browser-only machinery.

Pass    → Markdown is less than 50% of HTML size
Partial → Markdown is 50–80% of HTML size
Fail    → Markdown is more than 80% of HTML size

This check often catches the verbatim-HTML-to-Markdown conversion pattern: the response looks like Markdown at a glance, but the file size tells the real story.

Step 4: Aggregating the site score

Once every page has a 0–100 score, we compute a single site score. The home page is weighted 2x. Everything else is weighted 1x.

const HOME_PAGE_WEIGHT = 2;

function computeSiteScore(pageScores: PageScore[]): number {
  let totalWeight = 0;
  let weightedSum = 0;

  for (const page of pageScores) {
    const weight = page.isHomePage ? HOME_PAGE_WEIGHT : 1;
    weightedSum += page.score * weight;
    totalWeight += weight;
  }

  return Math.round(weightedSum / totalWeight);
}

Why double the homepage? It is the most likely starting point for exploration, and it is the only page eligible for the Sitemap / Index check. A broken root response should hurt more than one broken article.

The final score maps to one of three readiness states:

  • 70+: Agent-ready
  • 40–69: Partial support
  • Below 40: Not ready

What the score tells you

A score of 100 means the tested pages respond to the agent-style Accept header with clean Markdown, useful metadata and links, very little browser chrome, and a much smaller payload.

A score of 0 means every request returns the same thing regardless of Accept, the response body is raw HTML, and agents are getting the worst possible experience.

Most sites land somewhere in the middle. Those results are often the most actionable: the server is doing something, but one missing detail is wasting most of the benefit. Passing Markdown Response and failing Navigation Stripped, for example, usually means the format changed but the editing did not.

Run your own audit

The best way to understand the score is to give it a real site. Run an audit and you will get the seven-check breakdown, page-level evidence, and remediation guidance for anything that fails.

It is free. No signup. And if you disagree with the score, good—the report shows its work.