Resources

Does llms.txt Actually Work?

The file is real, cheap, and worth adding. It is also the most oversold item in AI search right now. Here is what the evidence says, and what actually decides whether an AI engine cites you.

By Sean Garschi, Founder, Growthr · Published August 26, 2026

llms.txt is a plain-text file at the root of a website that gives AI systems a short, structured summary of what the site is, who it serves, and which pages matter most. It was proposed in 2024 as a convention rather than a standard. No major AI engine requires it, and none has published a ranking benefit for having one.

That last sentence is the part agencies selling llms.txt implementation tend to leave out. We publish one at growthr.com/llms.txt, we think you should too, and we still would not describe it as the thing that gets you cited.

What the evidence actually shows

The honest summary in 2026 is that llms.txt adoption has run well ahead of any demonstrated effect. No major AI engine documents parsing the file, none lists it in its crawler documentation, and none has described it as an input to how answers get sourced. The engines that read your site are reading your HTML, the same way they read everyone else's.

This does not make the file useless. It makes it a bet with a low cost and an unproven payoff, which is a reasonable thing to place and an unreasonable thing to charge a premium for. The closest historical analogue is schema.org markup in 2011: a machine-readable layer that had no confirmed ranking benefit on the day it shipped and became table stakes later. Add it, keep it accurate, and do not expect it to carry the program.

The four things that actually decide whether you get cited

Across the engines that matter, citation comes down to four conditions, in rough order of impact.

The engine can fetch your pages. This is the only one that is binary. If robots.txt blocks GPTBot, PerplexityBot, ClaudeBot, or Google-Extended, that engine cannot cite you no matter what else you do. The same applies at the CDN layer, where a bot-management rule can block AI crawlers without anything appearing in robots.txt.
The page answers the question in a self-contained passage, near the top. Engines extract passages, not pages. A definition that arrives in paragraph four, after the scene-setting, loses to a competitor who leads with it.
The claim carries a specific number or a named source. The Princeton-led generative engine optimization study presented at KDD 2024 tested nine on-page tactics and found that citing sources and adding statistics produced the largest visibility gains of the set, while keyword stuffing reduced visibility. Vague authority claims did nothing.
Other sites the engine trusts say the same thing about you. This is the one most sites underinvest in, and the only one that moves recommendations rather than citations.

Being cited and being recommended are not the same thing

This distinction is worth more than any file you can add to your server. Being cited means an engine found your page useful enough to quote. Being recommended means it put your company on the buyer's shortlist. They are governed by different inputs.

Citation is largely within your control: structure, clarity, specificity, freshness. Recommendation is mostly a reflection of what the rest of the web says about you, and it barely moves in response to anything you publish on your own domain. A site can become reliably cited and still never appear on a shortlist, because no review platform, forum thread, or independent roundup corroborates it.

When your citations rise and your shortlist placements do not, the gap is offsite, and no amount of additional publishing will close it.

What we found testing this across 19 queries

In August 2026 we ran a fixed set of 19 buyer-intent queries through ChatGPT, Perplexity, and Google AI Overviews on the same day, and logged which sources each answer was built from. Two results were consistent enough to plan around.

Not one answer to an "agency for X" query cited an agency's own website. Every source was a third-party roundup, a review platform, or a forum thread. The vendor pages the agencies had presumably optimized did not appear, including in answers that recommended those same agencies by name. If you sell a service and you are competing for "best X for Y" queries, the page you control is not the page the engine reads.

Google AI Overviews appeared on 18 of the 19 queries. The exception was a person-name query. Planning for AI answers as an occasional format is already out of date for commercial searches.

A third pattern showed up on our own pages and is worth naming because it is fixable. On two queries our site ranked in the organic results while the AI Overview was assembled entirely from other sources. Being indexed, ranking, and being cited are three different states, and the gap between the second and the third is almost always page structure rather than authority.

How to check your own site in twenty minutes

Fetch your robots.txt and look for the named AI bots. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended. A blanket Allow: / under User-agent: * does not cover them, because a named user-agent group ignores the wildcard block entirely.
Request a key page with an AI crawler's user-agent string and confirm you get a 200 and real HTML. A CDN bot rule can return 403 to crawlers while every browser test you run looks fine.
Read the first 60 words of your most important page. If they do not answer the question the page is named after, that is your highest-value fix.
Check that your FAQ schema matches visible text. Answers that exist only inside JSON-LD are a mismatch, and the engine reading your prose never sees them.
Search your own category and read the source list. Not the answer, the sources. That list is your actual competitive set, and it is usually a handful of independent roundups you have never pitched.

Free scanners for the machine-readable layer: seogeoscan.com checks SEO and GEO signals, and willaiseeme.com checks whether an agent can read your site. Both are Growthr projects and both are free.

Frequently asked questions

What is llms.txt?

llms.txt is a plain-text file at the root of a website that gives AI systems a short, structured summary of what the site is, who it serves, and which pages matter most. It was proposed in 2024 as a convention, not a standard. No major AI engine requires it, and none has published a ranking benefit for having one.

Does llms.txt improve AI search rankings?

There is no published evidence that llms.txt improves rankings or citation rates on its own. Crawl studies through 2026 found that most published llms.txt files received no observed requests from AI crawlers. Treat it as cheap protocol-layer registration, similar to adding schema.org markup in 2011, rather than as a lever that moves visibility by itself.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which paths they may fetch and is honored by every major crawler. llms.txt tries to tell AI systems what your site means, and is honored by no one in particular. robots.txt has a direct effect on AI visibility: if it blocks GPTBot, PerplexityBot, ClaudeBot, or Google-Extended, those engines cannot cite you at all.

What actually makes an AI engine cite your site?

Four things, in rough order of impact: the engine can fetch your pages, the page answers the question in a self-contained passage near the top, the claim carries a specific number or source, and other sites the engine trusts say the same thing about you. The fourth is the one most sites underinvest in and the only one that reliably moves recommendations rather than citations.

Do I need llms.txt if I already have schema markup?

Schema markup does more for you than llms.txt does, because search engines actually parse it and document how they use it. If you have limited time, put it into schema, crawlability, and answer-first page structure first. Add llms.txt afterward, because it costs an hour and may matter later.

Want this run on your site?

A scored audit of your SEO and AI search standing, a prioritized fix plan, and the implementation. Flat fee.

See SEO + GEO pricing →