AI and search research # AI crawler readiness: llms.txt and WebMCP AI crawlers can read your site only if robots.txt allows them, key content renders without scripts, and, optionally, llms.txt points to it. Check those three first. A newer draft signal called WebMCP, which lets an AI agent act on a page, is early and optional. None of this guarantees an AI recommendation. ![](https://rewalto.com/_next/image?url=%2Fbrand%2Fog%2Fguides-ai-crawler-readiness.png&w=3840&q=75) In this guide 1. [What AI crawler readiness means](#what-it-means) 2. [The basics a crawler looks for, in order](#the-basics) 3. [robots.txt and llms.txt](#llms-txt) 4. [Content a crawler can actually read](#renderable-content) 5. ["Ready for AI agents" (WebMCP): an early, draft signal](#webmcp) 6. [Checking where your own site stands](#checking-your-site) 7. [Common gaps, in plain terms](#common-gaps) 8. [Reading is not the same as being trusted](#beyond-crawling) 9. [Who usually owns fixing a crawler-readiness gap](#who-fixes-it) 10. [What crawler readiness does not promise](#not-a-ranking-signal) 11. [Common questions](#questions) [Ask Finder](https://rewalto.com/finder) By [Varun Mahajan](https://rewalto.com/about), FounderPublished 30 September 2026Last updated 18 October 2026Part of the [ai visibility](https://rewalto.com/guides/ai-visibility-check) guide ## What AI crawler readiness means Search engines and AI assistants both send software to read your website before they can name it, describe it or send a shopper to it. AI crawler readiness is whether your site actually gives that software something it can read: pages that render as text, files that state what may be fetched, and structured data that says plainly what a page is about, rather than leaving it to be inferred. A page can look complete to a human visitor and still be nearly empty to a crawler, most often when its content only appears after a script runs. A crawler that does not execute that script sees a blank page, and a brand that never checks this has no way of knowing. ## The basics a crawler looks for, in order The diagram below sets out roughly the order a crawler, or an AI agent acting on a shopper's behalf, works through a site: what it fetches first, and what each later step needs from the site to succeed. Step 1 Fetch robots.txt States what may be crawled, and where the sitemap and llms.txt are. Step 2 Read llms.txt A plain-text map of the pages worth reading, for assistants that support it. Step 3 Render or read the HTML Text that only appears after a script runs may not be seen by a reader that does not run one. Step 4 Read structured data Schema.org markup states what a page is, in a form software can parse without guessing. Step 5 Check for an agent-ready endpoint An early, draft-standard signal (WebMCP) that a site offers a structured way for an AI agent to act, not just read. The order a crawler or an AI agent works through a domain, and what each step needs from the site. ## robots.txt and llms.txt robots.txt is the long-standing file that tells any crawler what it may fetch, and can point to a sitemap. llms.txt is a newer, plain-text convention some AI systems support: a short, readable map of the pages on a site worth reading, in a format meant for a language model rather than a search index. Neither file guarantees an assistant will read or cite a page. A crawler cannot use a file it cannot find or access. Check the [free utilities](https://rewalto.com/tools) for the current status of tools that can help prepare these files. ## Content a crawler can actually read The most common gap is content that exists only after JavaScript runs: a product description loaded by a script after the page appears, or text inside an image with no equivalent in the page's HTML. Whether a given crawler executes scripts varies and changes without notice, so the safer assumption is that some of the software reading your site will only ever see the HTML that arrives first. Structured data, Schema.org markup describing a product, an article or a business, gives a crawler a second, more reliable route to the same facts: not instead of readable text, but alongside it, stated in a form software can parse without guessing. ## "Ready for AI agents" (WebMCP): an early, draft signal A newer check some site audits now include asks whether a site offers a structured way for an AI agent to act on it, not just read it, sometimes described under the name WebMCP. Where a page tells a crawler what a product is, a WebMCP-style endpoint is meant to let an agent do something with that information in a defined, machine-readable way, rather than parsing a page built for a person. This is worth naming and worth tracking, and it is early. The standard is still a draft, support across AI agents is inconsistent, and a site with no such endpoint today is not doing anything wrong; it has no agent-facing endpoint yet. Treat a "ready for AI agents" result as a signal to watch rather than a requirement to meet immediately. ## Checking where your own site stands You can check the public files and pages yourself: inspect crawler permissions, confirm that the page’s main content is accessible, and review any structured data. Save the date and result of each check so you can compare it after a change. For the broader picture of how AI visibility differs from a crawler-readiness check, see [how to check whether AI assistants name your brand](https://rewalto.com/guides/ai-visibility-check). A crawler check reads your own site; an AI visibility check reads what assistants say about you. ## Common gaps, in plain terms The most frequent failure is not a missing file but a page that renders correctly for a person and shows nothing to software that does not run scripts: a product name and description injected after the page loads, with the initial HTML carrying only a loading placeholder. A brand that built its site to look good in a browser can still fail this test, because the test is about what arrives before any script runs, not about what a visitor eventually sees. A second frequent gap is inconsistency rather than absence: robots.txt exists but blocks paths a crawler needs, or llms.txt exists but lists pages that no longer exist or omits ones that matter. A file that is present but wrong can be worse than one that is missing, because it looks like the job was done. A third is structured data that describes a page inaccurately, most often because it was added once and never updated as the page's content changed. Schema.org markup that disagrees with the visible page is a signal a crawler can reasonably discount, so keeping the two in step matters as much as adding the markup in the first place. ## Reading is not the same as being trusted A crawler that can technically read a page is not the same as an assistant that will cite it. Readability removes a barrier; it does not create a reason to be chosen over another page that says something similar. That reason usually comes down to the same things that make a page useful to a person: it answers a specific question clearly, it is consistent with what the same brand says elsewhere, and it is not the only place making the claim. Treat crawler readiness as the floor, not the ceiling. A site that passes every technical check and says little of substance is still a thin source. A site that says something clear and specific, and is also readable, is in the best position of the three. For how the two connect, see [which sources AI assistants cite](https://rewalto.com/guides/ai-cited-sources). ## Who usually owns fixing a crawler-readiness gap Most of what a crawler-readiness check finds sits with whoever maintains the website: a development or web team can usually add or correct robots.txt and llms.txt, and can check whether core content survives with scripts disabled. Structured data sometimes sits with the same team and sometimes with whoever manages product content, since a Schema.org description of a product needs to track the product page itself. The WebMCP-style agent-readiness signal is newer territory, and at most organisations nobody yet owns it, because the underlying standard is still a draft and adoption is early. That is a reasonable state to be in today. The useful move is not to build an agent-facing endpoint immediately, but to know the check exists, know roughly what it is watching for, and revisit it as the standard and the tools around it mature. ## What crawler readiness does not promise Passing every check here does not promise a mention, a citation or a ranking from any search engine or AI assistant. It removes a class of failure, a page a crawler cannot read at all, and stops there. Search engines and AI assistants both make their own decisions about which readable pages to surface, based on material well beyond what this guide covers, and no outside party can guarantee that outcome. Be cautious of anyone who claims a specific technical fix will produce a specific ranking or citation result. The claim you can support is narrower: a page that can be read has a chance to be considered, and a page that cannot be read has none. ## Common questions ### What is AI crawler readiness? Whether your website gives search engines and AI crawlers something they can actually read: renderable content, a correct robots.txt and llms.txt, and structured data. ### What is llms.txt? A proposed plain-text format for describing a site and pointing readers to useful pages. It is separate from robots.txt and does not control crawler access or guarantee that an AI system will use it. ### What is the "ready for AI agents" (WebMCP) check? An early, draft-standard signal for whether a site offers a structured way for an AI agent to act on it, not only read it. Support is inconsistent today, so treat it as a signal to watch rather than a requirement. ## Start with the question you need to answer Check the current data catalogue, then review the coverage and dates against your question. The research in an older guide does not establish that a service is available today. [Explore Rewalto data](https://rewalto.com/)[Explore current data](https://rewalto.com/data) ## Related guides [AI and search research ### How to check whether AI assistants name your brand How assistants come to name brands, why a single answer misleads, and a first check you can run by hand. Explore](https://rewalto.com/guides/ai-visibility-check) [AI and search research ### Which sources AI assistants cite, and how to find yours Why there is no universal list of cited sources, and a method to find the ones behind your brand's answers. Explore](https://rewalto.com/guides/ai-cited-sources) --- Canonical page: https://rewalto.com/guides/ai-crawler-readiness Last content change: 2026-10-18