Does your website need llms.txt, and what is it?
Your site does not need llms.txt for ChatGPT or Alice, Yandex's AI assistant. It is a text file where a site describes itself to AI models. But AI models take the sites for their answers from search. In two weeks, their bots did not open our llms.txt on kaufmanlab.ru even once. If you don't have the file, there is no rush. First check in Yandex Webmaster (Yandex's free tool for site owners) whether your site is in search.
Roman Kaufman, founder of Kaufman Lab · Data as of: September 2026
Our llms.txt has been on the site since the first version. AI bots do visit us. In the second half of September, ChatGPT's search bot opened robots.txt, the file with rules for bots, 202 times. It did not look at llms.txt once. And when we reread our llms.txt ourselves, we found three lines in it that said untrue things about us. Even our own site check missed them.
AI models take sites for answers from search
Yandex says it directly: answers from Alice AI are "built on search results". Because of this, there are no new requirements for sites. ChatGPT has its own search and its own bot for it, OAI-SearchBot. This is stated in OpenAI's help for site owners.
Google employee John Mueller explained in the company's podcast in June 2026 why llms.txt does not help here. In essence, a site writes in it "my site is the best in the world", and there is no way to verify what a site says about itself. So first check whether your site is in search.
Who opened our llms.txt 60 times in two weeks
From September 16 to 30, 2026, our robots.txt was opened 887 times, and llms.txt 60 times. This is what the server log shows. It records which program opened which address and when. Here is who came for llms.txt.
- 15 times: YaCy, a small search engine;
- 13 times: our own site check bot;
- 10 times: scanners from website promotion services;
- about 20 times: people in an ordinary browser;
- 4 times: other programs;
- 0 times: bots from ChatGPT, Claude and Perplexity.
A large study shows the same picture. In June 2026, Ahrefs checked 137,210 sites. 38,360 of them had llms.txt, and 97% of these files were not requested by anyone in May.
The file was designed for assistant programs that do a task on a site instead of a person. According to the format description on llmstxt.org, such a program opens llms.txt and uses its links to find what it needs. According to Ahrefs, Claude Code, an assistant for programmers, fetched the file more often than any AI search bot. All AI search bots together made only 1.1% of requests to llms.txt.
The site writes its own description, and no one checks it
In llms.txt, a site describes itself, and no one outside will catch a mistake in that description. Our file is generated automatically, and before publishing, our check compares only the number of projects in it. The number matched, so three wrong lines got through.
The file said we have parsers, but we have no such projects. It also described our open study, where we ask AI models 24 questions and see if they name us. According to the file, we ask four AI models each time, but the second time we asked three. And it said every study has 96 checks, but the second one had 71 answers.
We found these lines only when we read the file line by line. We fixed all three on September 30.
While only scanners and people open the file, there is no harm. But an assistant program, the very thing the file is made for, will take it at its word and tell a person about a line of work you don't have. Only you can notice such a line, by comparing the file with your pages.
Check your site in search before you write the file
AI models take sites from search, and unlike a site's description of itself, your place in search can be checked. Start there.
- Add your site to Yandex Webmaster with the "Add site" button on webmaster.yandex.ru and verify your rights, that is, prove the site is yours. Verification takes up to a day.
- Open the "Pages in search" report and find the pages you need with the status "In search". After the bot crawls them, pages can appear in search within two weeks. If a page you need is missing, find out why the page is not in search.
- Open your-site.com/robots.txt and look for this pair of lines:
User-agent: * Disallow: /Instead of the asterisk, it may say OAI-SearchBot, the name of ChatGPT's search bot. This pair blocks the whole site for the bot, and ChatGPT will not show the site in its search answers. Remove the block.
If you don't have the file, don't make one for AI answers. It is useful when assistant programs visit your site, for example if you have documentation for programmers. How to build it and what to do with a file from a generator is covered in the questions below.
Frequently asked questions
How is llms.txt different from robots.txt?
llms.txt does not block anything. It is a hint for programs about what matters most on the site. robots.txt decides which bots may visit the pages.
How do I write llms.txt?
The only required part of llms.txt is a heading with the site name after the # sign. Then come a short description after the > sign and sections with links to the main pages. The file goes at your-site.com/llms.txt. Ours starts with the line "# Kaufman Lab", then a sentence about the lab, then a "Projects" section with links.
A generator made my file. Should I keep it?
Keep the generator's file if everything in it matches your site. Compare the sections, services and every number with your pages. Fix or remove any line that is not on the site.