---
title: "Making a website AI agents can actually read"
description: "The llms.txt files, Markdown mirrors, MCP server and tests behind a 100/100 agent readiness score, and what each one is for."
canonical: https://past.dev/blog/website-agents-can-read
date: 2026-08-26
category: Engineering
authors: The past.dev team
---
# Making a website AI agents can actually read

More agents than people read this site. That is not a prediction, it is our traffic, and it changes what shipping a page means: the HTML is one rendering, and the version an agent can fetch, parse and quote is the product. This post lists what we built, what each piece is for, and what it scored.

## The scoreboard, first

Is Agentic scores past.dev 100/100. Ora scores it 76/100, grade B, around rank 1,160 of 51,000 scanned sites, and its remaining deductions are off-site items like package registries and search presence, not the site itself. Lighthouse now has an agentic-browsing category; every page scores 100 on it. None of these audits agree on a rubric, which is exactly why we run all of them.

## One source, two renderings

Every page has a Markdown twin. The docs render HTML and Markdown from the same typed content model, so they cannot disagree. On [the blog](/blog), the served `.md` file is the source file itself. This sounds like plumbing until it breaks: our docs landing was rewritten once while its mirror kept the old pitch, and every assistant that pressed the Copy page button got copy the site no longer showed. The fix was not a correction, it was a test: neither surface may hardcode prose, both derive from one module, and the build fails on drift.

## The surfaces, and what each is for

- [/llms.txt](/llms.txt) is the map: what exists, what it is for, and a Markdown link for every entry.
- /docs/llms-full.txt is the whole documentation in one fetch, about 15,000 tokens, for an agent that wants context rather than navigation.
- `Accept: text/markdown` on any page returns its mirror, and known AI crawlers get Markdown without asking.
- Machine URLs keep a file extension, and paths with an extension are exempt from rate limiting. An agent fetching text should never hit a bot wall.
- An MCP server at past.dev/mcp exposes `search_docs`, `read_page` and `list_pages`, so an assistant can query the docs as tools instead of scraping.
- The `.well-known` directory carries the catalogs agents check: the MCP manifest, an ARD ai-catalog with domain-anchored `urn:air` identifiers, an RFC 9727 API catalog, and an agent-skills index.
- robots.txt allows every AI crawler by name and states a Content Signals policy: search yes, ai-input yes, ai-train no.

## The trade we refused

Inlining CSS would raise the mobile Lighthouse performance score by about one point. It also doubles the homepage from 80 KB to 165 KB and pushes the first heading of a docs page to byte 77,000, behind a wall of style rules no agent cares about. We keep the CSS in a linked file and the h1 inside the first 19 KB of every page. When a human metric and an agent metric conflict, the reader that actually shows up wins.

## What a page must say

A reader that cannot see layout gets no meaning from position or color, so anything that matters is stated in words. Status lives in text, not in a chip. Exact identifiers stay exact: a prettified scope is a support ticket. And every claim an agent might quote carries its evidence inline, which is also just [how we publish benchmarks](/blog/measuring-memory-honestly).

## Where to start on your own site

Serve llms.txt, give long-form pages a Markdown twin from the same source as the HTML, and put the whole docs in one fetchable file. Then run the audits: they disagree with each other enough to catch different failures. The deeper background on why agents need structured memory at all is in [What is agent memory](/what-is-agent-memory), and the API this site exists to document is in the [quickstart](/docs/memory-api/quickstart).