ChatGPT Gave Us an SEO Audit Without Collecting the Evidence

I opened a fresh ChatGPT thread and asked it to audit axiondeepdigital.com. Four prompts. No setup, no priming.

I did not upload a Lighthouse report. I did not connect Search Console. I did not give it our crawler output or DeepAudit.

It still returned precise scores, named the technologies on our homepage, cited specific location URLs, drew conclusions about how Googlebot renders the site, and handed me a prioritized plan.

Several of those claims were false. But the mistakes are not really the story.

The answer looked measured when no measurement had happened.

First, the limits of this test

One conversation, one site, one afternoon. Four turns in a single thread, not four independent trials, so later answers could see earlier ones.

I also did not record the model version, the plan, or whether any browsing tool fired. That was sloppy of me, and it is the first thing I would fix if I ran this again, because without it nobody can reproduce what I saw. Take everything below as one documented incident, not a benchmark of ChatGPT.

The scores had no disclosed calculation

The first answer graded us: Technical SEO 9/10. Architecture 8.8/10. Content Depth 6.8/10. Topical Authority 6.5/10. Local SEO 8.5/10. Conversion SEO 8.7/10.

A custom rubric is not automatically wrong. We publish a proprietary score ourselves, and I will come back to that.

The problem is that nothing explained why Architecture was 8.8 rather than 7.8 or 9.2. No criteria, no weights, no evidence, no pages named, no scale. The decimal implies a calculation that never happened. It was a vibe wearing a lab coat.

Then it described a homepage it had not read

It listed eight technology brands on our homepage. Three of the eight are not on the page.

It cited four location-page URLs. Two of the four do not exist.

These are not subtle SEO judgments. They are page-level facts, trivially checkable, and it got them wrong while sounding completely certain.

The dangerous part is the mixture. The fabricated items sat beside real ones, and the real ones lent them credibility. A list that is 60 percent right reads as a list that was checked.

Two messages later, it admitted it had never measured anything

I asked for our current mobile LCP, CLS, and INP.

I cannot provide live CrUX field values without direct access to PageSpeed Insights or the CrUX API.

Correct, and useful. It also detonates the first answer. Two turns earlier it had assigned a technical score and praised our rendering stack without ever having run a test.

A qualitative opinion without Lighthouse is fine. A *measured* conclusion without measurement is not. The later honesty exposed how little was under the earlier confidence.

Then it guessed at the vitals anyway: CLS very good, LCP decent, INP "likely your weakest."

Where we made our own mistake

We ran Lighthouse mobile and published a table comparing its numbers with those guesses. That table was wrong, and it was wrong in a way that matters, so we are fixing it here rather than quietly deleting it.

We compared a predicted INP against a measured TBT. Those are not the same metric. INP is a real-user interaction metric. TBT is a lab measurement of main-thread blocking during page load. TBT can hint at responsiveness risk. It is not INP, and treating one as the other is exactly the kind of sloppiness this article is complaining about.

We also reported Lighthouse's weighted score contributions (LCP 2/25, TBT 27/30, CLS 25/25) as if they were metric values. They are not. They are how much each metric contributed to the overall Performance score. We never recorded the raw seconds and milliseconds, so we cannot show them to you now. That is our failure, not Lighthouse's.

What survives, honestly:

• LCP scored in the lowest tier of its weighted contribution on mobile. That is a real, measured problem, and ChatGPT called it "decent."
• CLS was strong, which ChatGPT predicted. It is also the safest guess available.
• We could not test the INP prediction at all. Our origin does not have public CrUX field data, and the Lighthouse page-load report does not measure real interactions.

So the honest verdict is narrower than the one we originally published. ChatGPT missed a measured LCP problem. We cannot claim its ranking was "exactly inverted," because we never measured the metric it ranked worst.

"Based on the rendered structure"

The third answer opened with that phrase and went on to describe how Googlebot handles our JavaScript.

Nothing in that conversation showed it had opened the page in a browser, executed the scripts, inspected the DOM, or compared any of it against Google's rendered result.

I want to be careful here, because our earlier version of this post said "ChatGPT cannot render anything," and that is no longer true of the product. ChatGPT can search the web, cite sources, and in some modes drive a visual browser. A custom agent can be wired to Lighthouse, Search Console, or a crawler.

So the rule is not "an LLM can never inspect a site." The rule is that the answer has to say what it actually did in this run. Was the page fetched? Was JavaScript executed? Which URLs were opened? Which claim came from which tool?

Without that provenance, "based on the rendered structure" is just a convincing phrase.

The priorities ignored the thing we could measure

The last answer recommended a topical cluster, portfolio case studies, and better internal semantic links.

None of that is bad advice. It is also interchangeable with the advice you would give any agency site on earth, and it never mentioned the one problem an actual measurement had surfaced.

A priority list should show its work: the evidence, the affected pages, the confidence, the expected impact, the effort, and how you will know it worked. Otherwise a familiar recommendation sounds specific while remaining generic.

I will resist the temptation to say "a real list starts with fix mobile LCP." We had one lab run, no field data, and no evidence connecting that number to a lost ranking or a lost lead. It deserved investigation and a place on the list. Whether it outranked everything else is a claim I have not earned either.

The standard has to apply to us too

We give our own site a 96/100. ChatGPT gave itself a 9/10 for our technical SEO. Both are proprietary composites.

The difference I want to claim is that ours traces to versioned, documented checks against evidence collected from a real browser. The difference I am *not* entitled to claim is that ours is therefore true. Our scanner has produced false positives on our own site, and an over-strict heading rule of ours inflated a number in several published studies before we caught it.

An automated tool that skips the measurement is not better than a language model that skips the measurement. It just fails more quietly.

What an LLM is actually good for here

It is not the measurement instrument. It is the layer that helps a person understand measurements that actually happened.

Give it a Lighthouse report and it will explain it well. Give it a crawl export and it will find the contradictions. Give it your initial HTML and your rendered DOM and it will tell you what changed between them. Give it verified findings and it will draft the fix.

Ask it for a number it never collected and it will produce one, because producing plausible text is what it does.

The tools that answer the measurement questions are the boring ones: PageSpeed Insights for the lab run and field data when your origin qualifies for it, Search Console's URL Inspection for what Google actually crawled and rendered, a browser crawler for the source-versus-rendered comparison, and a person for whether the page is any good.

The test I should have run

One site and one thread cannot tell you how often this happens.

The real experiment is controlled pages with known defects: a missing H1, an H1 injected by JavaScript, a broken bundle, a conflicting canonical, a slow hero, absent structured data, a city page that does not exist, and a clean control. Run each through an untooled model, a search-enabled one, an agent with a browser, and a real auditor. Then score every claim for correctness and support.

That answers the only question worth asking: does the diagnosis change when the site changes?

I suspect it often would not. I have not proven it, and I am not going to write the conclusion before I run the test.

The rule we use now

We stopped asking whether AI wrote the audit. We ask whether each conclusion traces to evidence.

A performance number cites the run that produced it. A missing element names the URL and which representation was inspected. A rendering claim states how the page was rendered. A priority explains why it beat the alternatives.

And when the evidence is not there, the correct answer is not a decimal score. It is: I do not know yet, and here is the test that would tell us.

Run the audit if you want the browser-collected version. Then check it, because that rule applies to our tool too.