Can AI Read Your AI Company's Site? We Scanned 100

In September 2026 we scanned 100 AI companies to see how well the industry closest to this technology does at being read by it. They came out above average and nowhere near the top. The sample covers the labs, image and video generation, applied AI in law and healthcare and support, coding tools, data labelling, inference and compute, vector databases, evaluation tooling, speech, and robotics. We picked the names by hand, so this is a convenience sample rather than a random one, and no company here is named as failing anything.

Each domain went through three passes.

1. Our free scanner ran 22 checks on whether a machine can fetch and parse the site.
2. A second pass read each sitemap.
3. A third pulled one real article out of the sitemap and looked for a date on it.

Median score 72.5, against 65 for online retail and 77 for business software. Two companies scored above 90 and the best managed 91.

They are the best in the series at the boring parts

AI companies keep better housekeeping than any industry we have scanned. Ninety-three percent serve a sitemap and 61% put dates in it saying what changed, both the highest we have measured. Sixty-one percent of their articles carry a date a parser can read, also the highest.

Most of this is the stack rather than the strategy. These are recent companies on modern frameworks, and sitemaps with real dates, canonical tags, and server-rendered pages are what those frameworks do without being asked. It is the same reason the young software companies in our B2B sample outscored the enterprises.

Ten of them turn AI crawlers away

Ten of the 100 refused a request carrying an AI crawler's name. Six challenged every automated client, not AI specifically, and four served a browser normally while returning a 403 to GPTBot and ClaudeBot with nothing in robots.txt about it. None of the 100 documented a deliberate decision to block.

It would be easy and wrong to write this up as hypocrisy. A company whose crawler reads the web is not the same team as the one that bought the bot protection, and a 403 to a crawler user-agent is almost always a default rule doing what defaults do. That is the point worth taking from it: if the companies building this technology are refusing its crawlers by accident, the odds that your own edge is doing the same without anyone deciding to are higher than you would like.

Filter your edge logs for the GPTBot, ClaudeBot, PerplexityBot, and Google-Extended user-agents and look at the status codes you returned. It takes a minute and it settles the question.

One in five is blank until the browser fills it in

Twenty-one of the 100 returned a homepage with nothing on it before JavaScript ran. That is better than crypto at 34% and online retail at 52%, and worse than business software at 11%.

The pattern is specific: it is mostly the product-led companies whose marketing site is built in the same framework as the app, with a hero that animates in and copy that arrives with it. It looks excellent and reads as empty. Docs pages are usually fine, which matters, because for most of these companies the docs are what a model actually quotes.

Half of them do not publish the file written for them

Forty-seven of the 100 serve no llms.txt, and 96 do not link one from their pages.

We should be careful about how much that means, because it is our own bet. No major AI engine has published a ranking benefit for llms.txt, and we have said so in writing. Treat it as cheap protocol-layer registration, not a visibility lever. Still, it is the file proposed specifically so that language models can read a site, and the companies building language models adopt it at about the same rate as everyone else. That says more about the file's traction than about the companies.

Ninety of them cannot tell an engine who they are

Ninety of the 100 are missing pieces of the structured block that states a company's contact details, address, and links to its verified profiles. Seventy-four percent are also missing at least one of an about, contact, or privacy page at a predictable URL.

For companies in a crowded market with similar names, this is a strange thing to skip. That block is how a site says, in a form a machine can check, that it is a specific company rather than a string that resembles four other startups. It costs an afternoon.

Where our checks are opinions rather than standards

The llms.txt numbers above and the 80% that do not serve markdown to clients asking for it are both Growthr bets on where agent tooling is going, not settled standards. Reading them as an industry failure would be dishonest.

The checks that matter are the ones with evidence behind them: whether the crawler gets a page, whether that page has anything on it, whether it carries structure and a date, and whether other sites say the same things about you.

What we would fix first

None of this decides whether a model recommends you, which is governed mostly by what other sites say about you. It decides whether you are eligible to be read at all.

Method and limits

We scanned in September 2026. The 100 domains were chosen by hand, so the sample is not random and the numbers describe these companies rather than the industry. Each domain got a 22-check scan, a sitemap read, and one article pulled from that sitemap. The scanner ran 22 checks at the time of this scan; a 23rd, covering sitemap dates, was added afterwards, so a score you run today will not match one quoted here. An article was found for 72 of the 100, the highest of any industry we have scanned, so every date percentage is out of 72 rather than 100. No domain rate-limited the scan, so all 100 are scored. Every block was re-tested by hand from a second network before we described it.

The sample is named below; the results are not attributed to it. Every domain we scanned is listed, so anyone can reproduce this, but no company is identified as passing or failing any individual check. The point of the exercise is the pattern, and a per-company scoreboard would be a pile-on rather than research.

Frequently asked questions

© 2026 Growthr. All rights reserved. · llms.txt · About · Privacy · Terms