Can AI Read Your Fintech Site? We Scanned 100
We ran 100 US fintech websites through the checks we run for clients, then read their sitemaps and one real content page from each. The dates were the surprise.
In September 2026 we scanned 100 well-known US fintech websites to see what a machine gets when it asks them for a page. The sample covers payments, banking and neobanks, lending, investing, crypto, insurtech, payroll and spend management, and the infrastructure companies that sell to all of them. We picked the domains by hand from the companies people in the industry would name, so this is a convenience sample rather than a random one, and no company here is named as failing anything.
Each domain went through three passes. Our free scanner ran 22 checks on whether a machine can fetch and parse the site. A second pass read each sitemap and looked for lastmod dates. A third pulled one real content page per domain out of the sitemap, a blog post or a guide rather than the homepage, and looked for a date on it.
That third pass produced the result we did not expect.
Most of these sites publish a date no machine can read
A homepage has no particular reason to carry a date, so testing freshness there would fail sites that are doing nothing wrong. Content pages are different. AI systems weight recency, and a guide that cannot prove when it was written is competing against guides that can.
We found and fetched a content page for 58 of the 100 domains. The rest either published no sitemap, ran no content section, or organized their URLs in a way our matcher did not recognize. Of those 58 pages:
| On the content page | Share of 58 |
|---|---|
| Shows a date a reader can see | 86% |
| Carries a date a parser can read | 59% |
| Shows a date with nothing machine-readable behind it | 29% |
Uses a time element with a datetime attribute | 14% |
Nearly a third of these pages print a date for the reader and hand the parser nothing. One post we checked carries a byline reading June 23, 2021, five years old, with no time element, no dateModified in its schema, and no article:modified_time. A model reading that page cannot tell whether it is current or five years out of date, so it has no reason to prefer it over anything else.
Fixing this takes minutes. Wrap the date you already display in <time datetime="2026-09-06">, and make sure datePublished and dateModified in your Article schema say the same thing the page says. On a templated blog that is one change to one template.
Half give crawlers nothing to go on
A sitemap tells a crawler which URLs exist. The optional lastmod element tells it which ones moved, which is how a crawler decides what deserves a re-fetch instead of a skip.
| Sitemaps | Share of 100 |
|---|---|
| Serve a sitemap at the standard location | 83% |
Carry any lastmod dates | 55% |
So 45 of the 100 publish nothing about what changed and when. The ones that do keep it honest: of the sites with lastmod, 95% had touched something in the previous 30 days, and only 2% had a newest entry more than a year old. That matters, because a hardcoded date that quietly goes stale is worse than no date. Emit it from the CMS or the build, or leave it out.
Nobody scored above 89
The scanner ran 22 checks and weighted them by how much each one costs a site in retrieval. One domain rate-limited us on every request, so it has no score; scoring it zero would have measured its rate limiter rather than its markup. That leaves 99.
| Score across 99 sites | |
|---|---|
| Median | 72 |
| Best | 89 |
| Scored 90 or above | 0% |
| Scored under 70 | 40% |
| Scored under 50 | 11% |
These are companies with real engineering teams, and the gap is not on the marketing side of the stack. The heaviest check in the set asks whether the homepage carries real content before JavaScript runs, and 24% failed it. A page that is blank until it hydrates is a page most retrieval pipelines never see, because they do not run your JavaScript. A third had no valid JSON-LD anywhere on the homepage, and 91% had an Organization block missing the contact details, address, or linked profiles that let an engine confirm the company is who it says it is.
Eight of them block an AI crawler and their robots.txt does not mention it
This is the finding worth acting on, and it splits into three cases that need completely different responses.
One company blocks GPTBot and ClaudeBot on purpose. Its robots.txt names both and disallows them, and its server enforces it. That is a policy decision, someone made it deliberately, and it is reversible in one line whenever they want.
Eight companies return a 403 to an AI crawler while their robots.txt says nothing about it. A browser gets the page. Change the user-agent to GPTBot or ClaudeBot and the same URL returns 403. Nothing in robots.txt documents the decision, which is how you can tell it was probably made inside a bot-management console rather than by anyone thinking about AI search. Two of the eight block one crawler and allow the other, which is not a position anyone would choose on purpose.
Five challenge every automated client. These are worth naming as a separate case, because it would be easy to report them as AI-crawler blocks and be wrong. Bot-management products fingerprint the client through its TLS handshake and header order, not just its IP address and user-agent. We confirmed it on three of them: each returned 403 or 429 to a command-line request from a datacenter address and from an ordinary home connection, with a Chrome user-agent and with a GPTBot one, while a real browser loaded them without complaint. Sites in this group may still allowlist verified crawlers by published IP range, which nobody outside the company can check. Their own CDN logs answer it in a minute.
If you run one of these sites, the useful question is not whether your robots.txt allows GPTBot. It is whether your edge does.
Where our checks are opinions rather than standards
Two results look catastrophic and are not. 99% do not link an llms.txt file from their pages, and 93% do not serve markdown to clients that ask for it. Both are Growthr bets on where agent tooling is going, and we have written before that no major AI engine has published a ranking benefit for llms.txt. Reading a 99% failure rate on those as an industry problem would be dishonest. Nearly half of the sample publishes no llms.txt at all, which we would also not treat as urgent.
The checks that matter are the ones with evidence behind them: whether the crawler gets a page, whether that page contains content, whether the content carries structure and a date, and whether other sites say the same things about you.
What we would fix first
- Ask your CDN whether it is challenging AI crawlers. Filter your edge logs for the GPTBot, ClaudeBot, PerplexityBot, and Google-Extended user-agents and look at the status codes you returned. Eight of the companies here are almost certainly blocking without knowing.
- Serve real HTML. If your key pages are empty before hydration, server-side rendering for those routes buys more than everything below it combined.
- Mark up the dates you already show. A
timeelement and matching Article schema, on the template rather than per post. - Emit
lastmodfrom the build. Only if it will stay true. - Fill in the Organization block. Contact point, address, and the profile URLs that let an engine resolve you to a company rather than a string.
None of that decides whether a model recommends you, which is governed mostly by what other sites say about you. It decides whether you are eligible to be read at all.
Method and limits
We scanned in September 2026. The 100 domains were chosen by hand, so the sample is not random and the numbers describe these companies rather than the industry. Each domain got a 22-check scan, a sitemap read, and one content page pulled from that sitemap. The scanner ran 22 checks at the time of this scan; a 23rd, covering sitemap dates, was added afterwards, so a score you run today will not match one quoted here. A content page was found for 58 of the 100, so every freshness percentage is out of 58 rather than 100. One domain rate-limited the scan and is excluded from the scores. Blocks were re-tested by hand from a second network before we described any of them.
The sample is named below; the results are not attributed to it. Every domain we scanned is listed, so anyone can reproduce this, but no company is identified as passing or failing any individual check. The point of the exercise is the pattern, and a per-company scoreboard would be a pile-on rather than research.
The 100 domains we scanned
100 domains
stripe.com, plaid.com, squareup.com, paypal.com, marqeta.com, toasttab.com, bill.com, melio.com, tipalti.com, avidxchange.com, flywire.com, payoneer.com, remitly.com, moov.io, moderntreasury.com, lithic.com, highnote.com, unit.co, treasuryprime.com, synctera.com, column.com, increase.com, alloy.com, socure.com, withpersona.com, sardine.ai, sift.com, forter.com, bolt.com, affirm.com, sezzle.com, sofi.com, varomoney.com, current.com, dave.com, moneylion.com, albert.com, oportun.com, brex.com, ramp.com, mercury.com, rho.co, novo.co, bluevine.com, upstart.com, lendingclub.com, prosper.com, avant.com, fundbox.com, lendio.com, petalcard.com, missionlane.com, aven.com, earnin.com, branchapp.com, dailypay.com, robinhood.com, wealthfront.com, betterment.com, acorns.com, stash.com, public.com, webull.com, m1.com, titan.com, altruist.com, ellevest.com, monarchmoney.com, copilot.money, carta.com, coinbase.com, kraken.com, gemini.com, anchorage.com, fireblocks.com, circle.com, paxos.com, chainalysis.com, lemonade.com, joinroot.com, hippo.com, ethoslife.com, ladderlife.com, kin.com, nextinsurance.com, hioscar.com, better.com, blend.com, opendoor.com, gusto.com, rippling.com, deel.com, justworks.com, expensify.com, navan.com, pipe.com, capchase.com, nerdwallet.com, creditkarma.com, lendingtree.com
Frequently asked questions
How do I check whether AI can read my website?
Request your homepage with an AI crawler's user-agent string and confirm you get a 200 and real HTML. Then view source and check that your main claims exist in the raw HTML rather than arriving after JavaScript runs. Then check that your content pages carry a date in a time element or in Article schema. Our free scanner runs these checks on your domain and shows you which ones fail.
Does a 403 to a scanner mean AI crawlers are blocked?
No. Bot-management products fingerprint the client through its TLS handshake and header order, not just its IP and user-agent, so command-line tools get challenged while browsers load the same page. The test that separates the cases is to try the same request from a residential connection and with a browser user-agent. If everything is challenged, that is client fingerprinting rather than an AI-crawler decision, and the site may still allowlist verified crawlers by IP range. Your CDN logs settle it in a minute.
Do publication dates affect AI search visibility?
AI systems weight recency, so a page that cannot prove when it was written competes against pages that can. The date has to be machine-readable to count: a time element with a datetime attribute, or datePublished and dateModified in Article schema, matching whatever the page shows a reader. A date rendered as plain text is invisible to a parser.
What is lastmod in a sitemap and does it matter?
lastmod is an optional element in an XML sitemap giving the date a URL last changed. Crawlers use it to decide what to re-fetch, so a sitemap without it tells them nothing about what moved since their last visit. It only helps if it is true, which means emitting it from the CMS or the build rather than hardcoding a date that goes stale.
Should a fintech website publish an llms.txt file?
It costs an hour and no major AI engine has published a ranking benefit for it, so treat it as cheap protocol-layer registration rather than a visibility lever. Crawlability, server-rendered content, structured data, and machine-readable dates all matter more and are all better evidenced. We wrote up the evidence in Does llms.txt actually work?
Want this run on your site?
A scored audit of your SEO and AI search standing, a prioritized fix plan, and the implementation. Flat fee.
See SEO + GEO pricing →© 2026 Growthr. All rights reserved. · llms.txt · About · Privacy · Terms