What an AI crawler actually reads on 13 Connecticut agency sites

On 2 September 2026 we fetched the homepage, robots.txt and llms.txt of 13 Connecticut and Connecticut-serving marketing agencies. No JavaScript, one request each, exactly what a crawler that does not run a browser would get.

Here is what came back.

The one thing everybody gets right

11 of 13 had a canonical tag. 12 of 13 had a meta description. Nobody blocked the AI crawlers.

That last one surprised us and it is worth saying plainly, because we expected the opposite and went looking for it. Several of these sites run on platforms that ship a long list of AI user agents in robots.txt, and it reads at a glance like a block. It is not. The agents are listed and then the group falls through to the same path rules everyone else gets. No agency in this set is turning the answer engines away.

So the gap is not permission. Everyone is allowed in. The gap is what there is to read once they arrive.

What is missing

Three more returned a 200 at /llms.txt and served something that was not an llms.txt. Two of those were a copy of robots.txt. That is worth knowing if you are checking a competitor: a 200 is not a yes.

The number that matters most

Without JavaScript, the median homepage in this set had 1611 words and 40 links. The range was wide. The best had 4998 words and 160 links. One returned 114 bytes, zero words and zero links.

That last site looks fine in a browser. Everything is there. It is assembled by JavaScript after the page loads, and Googlebot will render it and cope. GPTBot, ClaudeBot, PerplexityBot and CCBot will not. They read the file that came off the server, and the file that came off the server was 114 bytes.

If you want one thing to check on your own site this week, it is that. Fetch your homepage without a browser and count the words.

curl -s https://yoursite.com/ | wc -c

If that number is small and your site is not small, the difference is what an answer engine cannot see.

Where we came out

We ran ourselves through the same script, and one of our numbers is worse than the median.

syntropicfoundry.com returned 746 words and 14 links without JavaScript. The median for the other 13 was 40 links. We are less than half of it.

We know why. This site had exactly two links in its raw HTML until the first of September, when we prerendered the components that build the header, the footer and the cards. That took it from 2 to 14 and from 1 reachable page to 24. It fixed the failure and stopped short of the standard, because the homepage still only links to the three service tiers and not to the eleven individual service pages underneath them.

That is being fixed, and it is a fair thing to have found in our own data. A firm that publishes a crawler audit and comes second on it in its own table would rather say that here than have you run the script yourself.

What we are not telling you

We did not measure rankings, traffic, or whether any of this moved a single lead for anybody. Structured data and a readable homepage are the conditions for being cited, not proof that citation happened.

Nobody has a clean before-and-after on AEO yet, including us. Anyone showing you one this year is showing you a correlation with a story attached.

The method, so you can repeat it

One GET per URL over HTTPS, a desktop Chrome user agent string, a 25 second timeout, no cookies and no JavaScript. Words counted after stripping script, style, svg and noscript and then all remaining tags. Links counted as unique href values in an <a> tag in the returned bytes. Structured data read as @type values in the raw HTML. An llms.txt was counted as real only if it returned a 200 and began with a markdown heading, which is what llmstxt.org specifies.

Four domains in the original list of 17 did not resolve or refused the request, and are excluded rather than counted as failures.

Run it against your own site before you take our word for any of it.

• AEO
• Connecticut
• research