Which Industries Can AI Read? We Scanned 640 Sites

Over September 6 and 7, 2026 we scanned 640 websites across six industries to find out which of them a machine can actually read. Every domain got the same checks from the same tool on the same days. Nobody scored above 92 except one outlier, and the spread between the best industry and the worst was 12 points of median score and nearly a factor of five on the failure that matters most.

Each of the six industries has its own write-up. This page is the comparison, and the short version is that the thing separating the top from the bottom is not budget, headcount, or how much anyone cares about SEO. It is how recently the site was built.

The table

Scores run 0 to 100 across 22 weighted checks. "Blank without JS" is the share whose homepage carried no real content until the browser ran the site's code. "Sitemap dates" is the share publishing dates that tell a crawler what changed. "Refuse AI crawlers" counts sites that turned away a request naming GPTBot or ClaudeBot, for any reason.

Age of the stack beats effort

We tested this directly. Alongside the 100 large B2B software companies, we scanned 40 seed-to-Series-B and independent ones with the same checks, expecting the well-resourced incumbents to win.

The small companies won, and the margin at the top was more than double. A site built in the last three years sits on a framework that emits server-rendered pages, canonical tags, sitemaps with dates, and valid structured data when you do nothing at all. A site built in 2016 sits on a stack that predates every one of those defaults, wrapped in years of tag managers and redirects, behind a security appliance somebody configured once and left.

The same pattern explains the industry table. AI companies keep the best sitemaps in the study not because they think hardest about crawlers, but because almost all of them are recent companies on current frameworks. Online stores come last partly because a storefront is the one kind of site that is still routinely assembled in the visitor's browser.

Which is worth saying plainly: a bad score is mostly a fact about when your site was built. That also makes it more fixable than it looks, because replacing a layer is a project you can scope, and persuading an organization to care is not.

The failure that decides the rest

One check carries more weight than any other, and it is the one asking whether the homepage has anything on it before JavaScript runs. A page that is empty until the browser fills it in is a page most retrieval pipelines never see, because they do not run your code.

Nothing else on any of these pages matters until this one passes. A store failing it is not competing badly in AI search, it is absent from it, and no amount of structured data changes that.

Everybody shows dates to people and hides them from machines

We pulled one real article from each site's sitemap and looked for a date on it. Almost every industry prints one for the reader and leaves the parser nothing, because the date is written as ordinary text that a machine cannot tell from any other sentence.

Online stores are the exception, and they are the exception for a reason that has nothing to do with skill. Hosted store software writes the publication date into the page's structured data automatically, whether or not the theme prints it on screen. It is the only place in this study where the default does the right thing without anyone deciding to.

Everywhere else the fix is the same and takes minutes: wrap the date you already display in a time element, and make the datePublished and dateModified in your structured data say what the page says.

Almost nobody can tell an engine who they are

The most consistent failure in all 640 sites was not a crawler problem or a rendering problem. Between 85% and 98% of every industry is missing pieces of the structured block stating a company's contact details, address, and links to its verified profiles.

That block is how a site says, in a form a machine can check, that it is a specific company at a specific address rather than a name that resembles four others. It is an afternoon of work, it is the same afternoon for every industry, and essentially nobody has done it.

The blocking is mostly an accident

One hundred of the 591 sites we could measure turned away a request naming an AI crawler. Only four of them documented the decision in robots.txt, and two of the four were medical content publishers whose business is people reading their articles, which is a coherent reason to say no.

The rest splits into sites whose security challenges every automated client, AI crawler or not, and sites that serve a browser normally and return a 403 to a crawler with nothing written down anywhere. That second group is the one worth acting on, because a real policy gets recorded and these were not.

It is worth being careful here. Bot-management products fingerprint a client through its TLS handshake and header order, not just its user-agent, so a scanner being refused is not proof that AI crawlers are refused, and a site may still allow verified crawlers by published IP range in ways nobody outside can test. We re-tested every block by hand from a second network before describing it. Your own edge logs settle the question in a minute: filter for the GPTBot, ClaudeBot, PerplexityBot, and Google-Extended user-agents and look at what you returned.

What we would fix first, in any industry

1. Send a finished page. If the homepage is empty before JavaScript runs, nothing below this line matters.
2. Ask what your edge returns to AI crawlers. Ninety-six of the 100 refusals in this study appear to be nobody's decision.
3. Mark up the dates you already show. A time element and matching structured data, on the template rather than per post.
4. Fill in the organization block. Contact point, address, verified profile links. The most consistent gap in the whole study.
5. Publish sitemap dates. Your build already knows them.

None of this decides whether a model recommends you over a competitor, which is governed mostly by what other sites say about you. It decides whether you are eligible to be read at all, and every industry here is leaving that on the table.

Method and limits

We scanned 640 domains over September 6 and 7, 2026: 100 in each of six industries, plus a 40-domain comparison cohort of small B2B software companies. Every list was chosen by hand, so none of them is a random sample, and the numbers describe these companies rather than their industries. Each domain got a 22-check scan, a sitemap read, and one article pulled from that sitemap and tested for a date. The scanner ran 22 checks at the time of these scans; a 23rd, covering sitemap dates, was added afterwards, so a score you run today will not match one quoted here.

Nine domains rate-limited the scanner and are excluded from the scores, which is why the industry counts sit at 97 to 100 rather than a flat 100. Date percentages are out of the number of sites where we found and fetched a real article, which ranged from 34 to 72 and is stated in each industry's write-up. A handful of domains overlap between samples, seven between crypto and fintech and three between B2B software and fintech, because those companies genuinely belong to both, so the studies are not independent samples of one another.

Each industry page lists its full 100-domain sample. No company anywhere in this study is identified as passing or failing any individual check.

Frequently asked questions

© 2026 Growthr. All rights reserved. · llms.txt · About · Privacy · Terms