Agent Ready: Every Check That Makes Your Site Usable by AI Agents

For most of the web's life there was one audience that mattered: people, arriving through a browser. That's changing fast. A growing share of visits now come from AI agents — ChatGPT browsing on a user's behalf, a coding assistant calling your API, an autonomous agent comparing products before it buys. These clients don't read your site the way a person does. They look for machine-readable signals: can I crawl this? is there a clean content map? does it expose tools I can call? how do I authenticate? can I pay?

WebScore's Agent Ready card (in the SEO module of every scan) checks your site against the emerging standards that answer those questions. This guide walks through every row on that card — what each one is, why it matters, and exactly how to add it.

The checks fall into four groups, from the most established to the most experimental.

Bot access & content

The foundation: can agents reach your content at all, and is it in a form they can use cleanly?

AI crawler access

The single most important check. Your robots.txt decides whether the crawlers that feed AI assistants — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others — are allowed to read your site. Block them (deliberately or by accident, via a copied robots.txt or a security plugin) and you're invisible to ChatGPT, Claude, Perplexity and Google's AI answers.

How to add it: in /robots.txt, add explicit User-agent sections for the AI crawlers and Allow the paths you want cited. Don't Disallow the content you want models to surface. (See our AI discoverability guide for the full crawler list.)

Content Signals

A blunt allow/deny is a crude tool. Content Signals — a robots.txt extension popularized by Cloudflare — let you declare how your content may be used rather than simply whether it can be fetched: search indexing yes, AI training no, AI input (answering a live question) yes, for example.

How to add it: add a Content-Signal: line to /robots.txt, e.g. Content-Signal: search=yes, ai-train=no, ai-input=yes.

llms.txt

An /llms.txt file (the llmstxt.org convention) is a Markdown map of your most important pages — your site's table of contents, written for models. Instead of guessing which of your pages matter, an assistant can read a curated list with a one-line description of each.

How to add it: publish a Markdown file at /llms.txt with your site name, a one-line summary, and links to your key pages (docs, pricing, top articles). Our step-by-step llms.txt guide has a copy-paste template.

Markdown responses

HTML is noisy — navigation, scripts, styling — and an agent has to wade through all of it to find your actual content. If your server can return clean Markdown when an agent asks for it (via the Accept: text/markdown request header), you hand the model exactly what it needs and nothing else.

How to add it: content-negotiate on the Accept header. When a request includes Accept: text/markdown, respond with Content-Type: text/markdown and a Markdown rendering of the page.

Web Bot Auth

The flip side of crawler access: how do you know a bot claiming to be ClaudeBot really is? Web Bot Auth lets legitimate bots cryptographically sign their requests (using HTTP Message Signatures), so you can distinguish real agents from spoofers and apply the right rules to each.

How to add it: publish your signing-key directory at /.well-known/http-message-signatures-directory and verify the Signature / Signature-Agent headers on incoming bot requests.

Agent discovery

Beyond reading your content, can an agent discover what your site can do — its APIs, tools and capabilities?

Link headers

RFC 8288 Link: response headers let an agent discover related resources straight from your HTTP response, before it even parses the body — a pointer to your API catalog, your auth metadata, your sitemap.

How to add it: emit Link: response headers pointing at your discovery resources, e.g. Link: </.well-known/api-catalog>; rel="api-catalog".

DNS-AID

DNS for AI Discovery publishes agent metadata in DNS itself, so an agent can find your endpoints before it ever fetches a page. Two conventions exist: the agentcommunity AID spec uses a TXT record at _agent.<domain>; the IETF draft uses an _agents.<domain> namespace.

How to add it: publish a TXT record at _agent.<yourdomain>, e.g. v=aid2;u=https://yourdomain/mcp;p=mcp;s=Your tools. (Note: it must be a real TXT record — an address-type wildcard, common on Cloudflare, does not count and shouldn't be reported as present.)

MCP Server Card

The Model Context Protocol (MCP) is the emerging standard for connecting AI assistants to external tools and data. An MCP Server Card advertises that your site runs an MCP server, so an agent can connect and use your tools directly.

How to add it: publish a server card at /.well-known/mcp.json describing your MCP endpoint and its capabilities.

API Catalog

RFC 9727 defines /.well-known/api-catalog — a machine-readable index of your APIs. Rather than reverse-engineering your endpoints, an agent reads the catalog and finds your OpenAPI descriptions.

How to add it: publish a catalog at /.well-known/api-catalog linking to your API descriptions.

Agent Skills index

An agent skills index enumerates the discrete actions an agent can perform on your site — "search products", "check order status", "book a slot" — and how to invoke each.

How to add it: publish a skills index (e.g. /.well-known/agent-skills.json) listing the actions agents can take.

WebMCP

WebMCP brings MCP into the browser: a site exposes its tools directly to in-page agents (through the navigator.modelContext browser API or a declarative MCP script tag), with no separate server to run.

How to add it: register your tools with the WebMCP browser API, or declare an MCP script tag so in-browser agents can call them.

Agent authentication

Discovery isn't enough if an agent can't get in. These checks tell an agent how to authenticate to your APIs — automatically.

OAuth/OIDC discovery

Publishing OAuth Authorization Server metadata (RFC 8414) or OpenID Connect discovery metadata lets an agent learn your authorization endpoints and how to obtain a token — without a human reading your docs.

How to add it: publish /.well-known/oauth-authorization-server or /.well-known/openid-configuration describing your auth endpoints.

OAuth Protected Resource

RFC 9728 Protected Resource Metadata connects your API to the authorization server that protects it, so an agent holding a resource knows exactly where to go for a token. It's the piece that makes MCP's authorization flow work end to end.

How to add it: publish /.well-known/oauth-protected-resource linking your API to its authorization server.

Auth.md

Not everything is machine-negotiated yet. Auth.md is a plain-Markdown document describing, for an agent (or the developer building one), how to register and authenticate with your service — a human-and-machine-readable README for your auth flow.

How to add it: publish /.well-known/auth.md (or /auth.md) describing your registration and authentication steps.

Agent commerce

The newest frontier: letting an agent pay you, per request, without a checkout flow built for humans. These protocols are early and still competing, but they point at where agentic commerce is heading.

x402 payments

x402 revives the long-dormant HTTP 402 Payment Required status code: a server responds 402 with payment requirements, the agent pays inline (via the X-Payment header) and retries — no account, no cart, no checkout page. Ideal for metered APIs and pay-per-use content.

How to add it: respond 402 with payment requirements and accept the X-Payment header, and/or advertise support at /.well-known/x402.

Machine Payment Protocol (MPP)

MPP is another approach to agent-native, per-request HTTP payments between machines, advertised through a discovery document.

How to add it: advertise MPP support via /.well-known/mpp.json describing your payment endpoints and accepted methods.

Universal Commerce Protocol (UCP)

UCP aims to let agents pay for content and products through a standard, cross-vendor flow — so a buyer's agent isn't rebuilt for every store.

How to add it: publish UCP discovery metadata (e.g. /.well-known/ucp.json) exposing your catalog and payment endpoints.

Agentic Commerce Protocol (ACP)

ACP exposes your commerce API so an agent can browse, add to cart and check out on a buyer's behalf — the full purchase, agent-driven.

How to add it: publish ACP discovery metadata (e.g. /.well-known/acp.json) describing your product feed and checkout API.

Where to start

You don't need all eighteen. Think of it as a ladder:

1. Every site: allow AI crawlers, publish an llms.txt, and consider Content Signals and Markdown responses. This is table stakes for showing up in AI answers.
2. APIs & SaaS: add the discovery and authentication groups — an MCP server card, an API catalog, OAuth discovery — so agents can find and call you.
3. Stores: watch the commerce protocols. They're early, but agent-driven buying is coming, and being first in a standard is cheap now and expensive later.

None of this changes what your human visitors or Google see. It only opens a second front door — for the agents that are already knocking.

Want to know where your site stands? Run a free WebScore scan and open the Agent Ready card in the SEO report to see which of these you already pass, and exactly how to add the rest.