By Varun Mahajan, FounderPublished Updated Part of the ai visibility guide4 min read
Key takeaways
- llms.txt is a proposal, not a formal standard, and no file can promise what an assistant says.
- robots.txt decides which crawlers may fetch your pages. An accidental block is the mistake worth catching.
- The earlier audit checked crawler access and page markup; that service is no longer offered.
- Allowing crawlers removes an obstacle. It does not create a recommendation.
This post is the timely take. The evergreen how-to is the guide Which sources AI assistants cite, and how to find yours, which covers the topic in full.
What is llms.txt, and is it a standard?
llms.txt is a proposal: a plain-text file at the root of your site that gives a language model a short, readable summary of what the site is and which pages matter. It is written for a machine that would rather read one tidy page than crawl fifty untidy ones. It is a convention someone suggested and some sites have adopted. It is not a formal standard, and no file can promise what an assistant will say.
We publish one ourselves, at rewalto.com/llms.txt, and we do not claim it gets us cited. We publish it because a missing file is an avoidable gap for a company that measures AI visibility, and because the Markdown copies of our pages that it links are easy for tools to read.
What does robots.txt have to do with AI crawlers?
robots.txt is the older and more important file. It tells any crawler which parts of your site it may fetch, and well-behaved crawlers read it first. The companies that run AI systems publish the names their crawlers go by, so you can allow or refuse each one by name. The free site audit looks for five of them: GPTBot, PerplexityBot, ClaudeBot, Google-Extended and CCBot.
Whether to allow them is your decision, and blocking one can be a deliberate choice. The mistake worth catching is the accidental block: a blanket rule written years ago for another reason, or a copy of a competitor's file that refuses everything. If a crawler cannot fetch your pages, it cannot read what you wrote about your own products.
What should you check on your own site for AI readiness?
Three things, and you can check each one yourself. Whether the five AI crawlers above are allowed in your robots.txt. Whether an llms.txt file is present. And whether your pages carry FAQ or question markup, which gives a machine a ready-made question and answer to lift.
Rewalto's earlier free site audit ran these checks; it has since been withdrawn. Open your robots.txt and llms.txt in a browser, and look at one key page's source for question markup. A check you could not run is a gap to note, not a pass or a failure.
How can I check my own files in five minutes?
You need a browser and nothing else. Open your site's address followed by /robots.txt and read it, then do the same with /llms.txt.
If /llms.txt does not exist yet, write one by hand from your site's own real key pages rather than from a generic template. Rewalto's earlier generator for this is withdrawn.
- In robots.txt, look for the five crawler names above and for a blanket Disallow: / under User-agent: *. Either can shut AI crawlers out. Ask whoever wrote the rule why it is there before you change it.
- In llms.txt, check that the page loads rather than returns not found, that it says in a few sentences what the site is, and that every page it links still exists.
- Then ask the same question of your product pages: is the fact a shopper wants, such as ingredients, pack size or price basis, stated in plain words that a machine can lift?
- Write down what you found and the date, so the next check is comparable.
If I allow the crawlers, will AI assistants cite me?
No. Allowing a crawler removes an obstacle, and it does not create a recommendation. An assistant may read your pages and still name someone else, or answer without a source at all. Technical readiness and observed AI answers are different kinds of evidence, and the audit keeps them apart for that reason.
To see what assistants say about your category, run the questions your shoppers ask and save the answers. The guide on where AI answers get their sources explains what assistants draw on, and the guide on checking whether AI assistants name your brand shows how to do it by hand.
Quick answers
Do I need an llms.txt file?
It is optional and unproven as a lever. It costs little to publish and it gives tools a tidy summary of your site. Do not expect it to change what an assistant recommends.
Should I block AI crawlers?
That is a business decision. Some publishers block them on purpose. If you sell products and want to be read, check that you have not blocked them by accident.
Which AI crawlers should robots.txt name?
GPTBot, PerplexityBot, ClaudeBot, Google-Extended and CCBot were the crawler names covered by this article’s checks. You can inspect your own robots.txt and compare it with each provider’s current documentation.