AEO # llms.txt and AI crawlers: what to check on your own site llms.txt is a proposed plain-text file that summarises your site for AI systems, and robots.txt is the long-standing file that says which crawlers may fetch your pages. Check that you have not blocked AI crawlers by accident and that both files load. Google says you can ignore llms.txt for Google Search, and neither file promises an AI recommendation. ![](https://rewalto.com/_next/image?url=%2Fbrand%2Fog%2Fblog-llms-txt-and-ai-crawlers.png&w=3840&q=75) Illustration ยท Illustrative editorial cover, not observed data. In this post 1. [Key takeaways](#takeaways) 2. [What is llms.txt, and is it a standard?](#what-is-llms-txt) 3. [What does robots.txt have to do with AI crawlers?](#robots-and-ai) 4. [What should you check on your own site for AI readiness?](#what-to-check) 5. [How can I check my own files in five minutes?](#check-by-hand) 6. [If I allow the crawlers, will AI assistants cite me?](#does-it-get-me-cited) 7. [Quick answers](#questions) 8. [Sources](#sources) [Ask Finder](https://rewalto.com/finder) By [Varun Mahajan](https://rewalto.com/about), FounderPublished 8 September 2026Updated 31 October 20265 min read ## Key takeaways - llms.txt is a proposal, not a formal standard, and no file can promise what an assistant says. - robots.txt decides which crawlers may fetch your pages. An accidental block is the mistake worth catching. - For Google Search, Google says you can ignore llms.txt. - Allowing crawlers removes an obstacle. It does not create a recommendation. This post is the timely take. The evergreen how-to is the guide [Which sources AI assistants cite, and how to find yours](https://rewalto.com/guides/ai-cited-sources), which covers the topic in full. ## What is llms.txt, and is it a standard? llms.txt is a proposal: a plain-text file at the root of your site that gives a language model a short, readable summary of what the site is and which pages matter. [Jeremy Howard published the proposal](https://llmstxt.org/) in September 2024. It is a convention some sites have adopted. It is not a formal standard, and no file can promise what an assistant will say. Google has said where it stands for its own search. Its [guide to generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), dated 10 July 2026 by Google, says that for Google Search you can ignore unnecessary AI text files such as llms.txt. So do not publish one to do better on Google. We publish one ourselves, at rewalto.com/llms.txt, and we do not claim it gets us cited. It lists every page in our sitemaps, each with a Markdown copy. We publish it because AI agents are among the readers of our data pages, and the Markdown copies it links are easy for tools to read. ## What does robots.txt have to do with AI crawlers? robots.txt is the older and more important file. It tells any crawler which parts of your site it may fetch, and well-behaved crawlers read it first. The companies that run AI systems publish the names their crawlers go by, so you can allow or refuse each one by name. The names do different jobs, as OpenAI, Anthropic, Perplexity and Google documented them when we read their pages on 31 October 2026. Whether to allow them is your decision, and blocking one can be a deliberate choice. [OpenAI says](https://developers.openai.com/api/docs/bots) its settings are independent, so a site can stay in search results and still refuse training. The mistake worth catching is the accidental block: a blanket rule written years ago for another reason, or a copy of a competitor's file that refuses everything. If a crawler cannot fetch your pages, it cannot read what you wrote about your own products. - Search: [OAI-SearchBot](https://developers.openai.com/api/docs/bots), [PerplexityBot](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) and [Claude-SearchBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) gather pages for each company's search results. Blocking one can cost you visibility there. - Training: GPTBot and ClaudeBot collect content that may be used to train AI models. Blocking them says you do not want your pages used that way. - A person's request: ChatGPT-User, Claude-User and Perplexity-User visit a page when someone asks a question. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person started the visit. - Google-Extended: not a crawler. [Google describes it](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) as a robots.txt control over whether crawled content may train future Gemini models and support grounding in Gemini Apps. Google says it does not affect a site's inclusion in Google Search. - CCBot: the crawler of [Common Crawl](https://commoncrawl.org/ccbot), a public archive of the web. ## What should you check on your own site for AI readiness? Three things, and you can check each one yourself. Whether the crawlers you want are allowed in your robots.txt. Whether an llms.txt file is present, if you chose to publish one. And whether a key page states its facts in plain words on the page. Open your robots.txt and llms.txt in a browser, and read one key page as a stranger would. A check you could not run is a gap to note, not a pass or a failure. ## How can I check my own files in five minutes? You need a browser and nothing else. Open your site's address followed by /robots.txt and read it, then do the same with /llms.txt. If /llms.txt does not exist yet and you want one, write it by hand from your site's own real key pages rather than from a generic template. - In robots.txt, look for the crawler names above and for a blanket Disallow: / under User-agent: \*. Either can shut AI crawlers out. Ask whoever wrote the rule why it is there before you change it. - In llms.txt, check that the page loads rather than returns not found, that it says in a few sentences what the site is, and that every page it links still exists. - Then ask the same question of your product pages: is the fact a shopper wants, such as ingredients, pack size or price basis, stated in plain words that a machine can lift? - Write down what you found and the date, so the next check is comparable. ## If I allow the crawlers, will AI assistants cite me? No. Allowing a crawler removes an obstacle, and it does not create a recommendation. An assistant may read your pages and still name someone else, or answer without a source at all. Technical readiness and observed AI answers are different kinds of evidence, so keep them apart. To see what assistants say about your category, run the questions your shoppers ask and save the answers. The guide on [where AI answers get their sources](https://rewalto.com/guides/ai-cited-sources) explains what assistants draw on, and the guide on [checking whether AI assistants name your brand](https://rewalto.com/guides/ai-visibility-check) shows how to do it by hand. If you build agents rather than run a brand's site, the same rule applies to what they read: a figure needs its date. Our [data catalogue](https://rewalto.com/data), [commerce data](https://rewalto.com/data/commerce) and [Dark-store map](https://rewalto.com/data/dark-stores) show the date with each reading, and the [developer docs](https://rewalto.com/developers) cover the API. ## Quick answers ### Do I need an llms.txt file? It is optional and unproven as a lever. It costs little to publish and it gives tools a tidy summary of your site. Do not expect it to change what an assistant recommends. ### Does Google use llms.txt? Not for Google Search. Google's guide to generative AI features, dated 10 July 2026, says you can ignore unnecessary AI text files such as llms.txt for Google Search. ### Should I block AI crawlers? That is a business decision. Some publishers block them on purpose, and some allow search crawlers while refusing training crawlers. If you sell products and want to be read, check that you have not blocked them by accident. ### Which AI crawlers should robots.txt name? Only the ones you want to treat differently from your default rule. The usual names are OAI-SearchBot, GPTBot, PerplexityBot, Claude-SearchBot, ClaudeBot and CCBot, plus the Google-Extended control. Compare your file with each provider's current documentation, because the names change. ## Sources 1. [Jeremy Howard: The /llms.txt file](https://llmstxt.org/), published 3 September 2024. The proposal itself and its date. 2. [Google: Optimizing your website for generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), published 10 July 2026. For Google Search, unnecessary AI text files such as llms.txt can be ignored. 3. [Google: Common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), published 30 October 2026. Google-Extended is a robots.txt control, not a crawler, and does not affect inclusion in Search. 4. [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots), published 31 October 2026. What OAI-SearchBot, GPTBot and ChatGPT-User do. The page carries no date; read on 31 October 2026. 5. [Anthropic: Crawler help article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), published 31 October 2026. What ClaudeBot, Claude-User and Claude-SearchBot do. Read on 31 October 2026. 6. [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), published 31 October 2026. What PerplexityBot and Perplexity-User do. Read on 31 October 2026. 7. [Common Crawl: CCBot](https://commoncrawl.org/ccbot), published 31 October 2026. CCBot is Common Crawl's crawler. Read on 31 October 2026. ## Keep reading - The full guide ### [Which sources AI assistants cite, and how to find yours](https://rewalto.com/guides/ai-cited-sources) Why there is no universal list of cited sources, and a method to find the ones behind your brand's answers. - Guide ### [How to check whether AI assistants name your brand](https://rewalto.com/guides/ai-visibility-check) How assistants come to name brands, why a single answer misleads, and a first check you can run by hand. - Guide ### [Share of AI vs share of search: two measures, two questions](https://rewalto.com/guides/share-of-ai) How to measure how often assistants name your brand, and how that differs from search demand. - From the blog ### [Zero-click search in India: why sessions are no longer the whole scorecard](https://rewalto.com/blog/zero-click-search-india) Zane Marketing's August 2026 case that sessions alone are an unfair scorecard, the four lines it suggests, and the second record most scorecards still miss. ## What to do next Read current data and access options. [Explore Rewalto](https://rewalto.com/)[Current data catalogue](https://rewalto.com/data)[Dark-store map](https://rewalto.com/data/dark-stores)[Check coverage and freshness](https://rewalto.com/guides/check-dataset-coverage-and-freshness) [Ask our data a question](https://rewalto.com/finder) Free to ask, no account. --- Canonical page: https://rewalto.com/blog/llms-txt-and-ai-crawlers Last content change: 2026-10-31