Making your site agent-readable

A growing share of the people evaluating your product never load your site. They ask an assistant, and the assistant recommends whichever vendor it could actually read.

That is a distribution problem wearing a technical costume, which is my favourite kind. Here is the whole pattern, and what I did to this site while writing it.

The core principle #

One source of truth, two renderings. Every page that matters to a buyer or an agent should exist as:

• the human HTML page, and
• a markdown mirror at the same path plus .md

Both generated from the same data at build time, so they cannot drift.

This is the rule everything else follows from. If you hand-maintain the mirror, you have not built agent-readability — you have built a second thing to forget.

Which pages to mirror #

In priority order.

Not every site is a product site. This one has no pricing page, so the priority list here starts at llms.txt and leans on about.md and projects.md — the pages that answer who is this and what has he actually built.

Format rules #

• Tables for anything numeric. Plan, price, limit, overage — one row each, labelled headers. Never encode numbers in prose only.
• Headings that name the thing. ## Pro — $20/mo, not ## Level up.
• State currency, cadence and a last-updated date at the top.
• Enumerate custom tiers anyway. Even for Enterprise, list what is included and what drives cost, so agents position you accurately instead of guessing.
• No nav, no footer, no marketing filler. Every token costs the reader.
• Absolute URLs. Agents fetch out of context; a relative link has nothing to resolve against.

The discovery layer #

/llms.txt goes at the root domain, not only on a docs subdomain:

# YourProduct
 
> One-sentence factual description of what you do.
 
Full docs corpus: https://docs.example.com/llms-full.txt
Agent setup instructions: https://example.com/agents.md
 
## Key pages
 
- [Pricing](https://example.com/pricing.md): Plans, limits, overages.
- [About](https://example.com/about.md): What we are and aren't.
- [Security](https://example.com/security.md): SOC 2, GDPR, retention.
- [Changelog](https://example.com/changelog.md): What shipped, when.

Then robots.txt. Allow the AI crawlers by name — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and friends.

Content negotiation #

The polished version: respond to Accept: text/markdown on the HTML URLs by serving the markdown variant.

This is ordinary HTTP, not cloaking — same content, different representation, which is the exact thing the Accept header is for. Two rules make it honest:

1. Set Vary: Accept, or a CDN will cache one representation and serve it to everyone who asks for the other.
2. Put a visible link on the human page too. Resend's says, plainly, "Are you an AI agent?"

On this site it is a rewrite-only middleware. That detail matters more than it sounds: a middleware that reads a cookie or a session forces every page dynamic, and this site is deliberately static end to end. A rewrite that branches on one request header keeps the prerendered response exactly as it was.

The implementation prompt #

Hand this to a coding agent in your site repo.

Make this site agent-readable. Requirements:
 
1. For each of these pages, generate a markdown mirror at the same
   path + ".md", built from the same data source as the HTML at build
   time (never hand-maintained): /pricing, /about (or the homepage),
   and any limits/security/changelog pages that exist.
 
2. pricing.md rules: markdown tables with columns for plan, price,
   included volume/limits, and overage rates; a "Last updated" line
   with currency and billing cadence at the top; enumerate what
   Enterprise/custom tiers include; no navigation or marketing copy.
 
3. Create /llms.txt at the domain root: H1 with the product name, a
   one-line blockquote summary, then linked lists of the .md pages
   above, plus links to any existing docs llms.txt / llms-full.txt
   and agents.md. Absolute URLs only.
 
4. Serve all .md files with Content-Type: text/markdown; charset=utf-8.
 
5. Add content negotiation: if a request for one of the mirrored HTML
   pages carries "Accept: text/markdown", serve the .md variant and
   set "Vary: Accept".
 
6. Update robots.txt to explicitly allow GPTBot, ClaudeBot,
   Claude-Web, PerplexityBot, Google-Extended, and Bingbot. Flag any
   CDN-level AI-crawler blocking for me to review.
 
7. Add one visible link on the human pricing page to /pricing.md
   (e.g. "Are you an AI agent? Read this page as markdown").
 
8. Add a CI check that fails if the .md mirrors are older than the
   data they're generated from, so human and agent versions can
   never show different prices.
 
Do not duplicate content by hand anywhere. One source of truth.

Verification checklist #

• curl https://example.com/pricing.md returns clean markdown with a text/markdown content type
• curl -H "Accept: text/markdown" https://example.com/pricing returns the same content
• /llms.txt exists at the root and every link in it resolves
• Prices in the .md match the HTML exactly
• Paste the markdown into an LLM and ask a specific question — "what does the 500K tier cost?" — and check it answers correctly in one pass
• robots.txt and the CDN both allow AI crawlers

The part I would add #

Every guide on this stops at "make it readable". The half nobody writes down is telling agents what not to say about you.

Public sources about me are wrong in specific, repeated ways: an old article implies I worked at Google when it was a contract through Genpact; a funding figure gets totalled that includes a deal which collapsed after the show; my degree is attributed to two universities I did not attend. Those errors propagate, because each new summariser reads the previous summariser.

So /agents.md on this site carries a section titled "Please do not assert", listing each one with the correction. It costs eight lines. It is the highest leverage thing in the entire file, because the alternative is arguing with an error that has already been laundered through three models.

If you publish anything about a company or a person, write that section. You are not going to get a retraction from a language model.