Can AI Search Read Your Website? A Technical Checklist
Most AI crawlers do not run JavaScript. Here is how to check what ChatGPT, Perplexity and Google actually see on your site, and what to fix first.
Short answer: if your website shows its content only after JavaScript runs in the browser, most AI assistants cannot read it. A 2024 study by Vercel and MERJ found that none of the major AI crawlers, including OpenAI's, Anthropic's and Perplexity's, executed JavaScript. They saw the raw HTML your server sends, and on many modern sites that HTML is close to empty.
That matters more every month, because a growing share of buyers now ask ChatGPT, Perplexity, Gemini or Google's AI Overviews before they ever open a search result. If the assistant cannot read your pages, it cannot recommend you. This checklist covers what to verify and what to fix, in order of impact.
Why AI crawlers see less than your visitors do
When a person opens your site, their browser downloads the HTML, runs your JavaScript, calls your APIs and paints the final page. A crawler that does not run JavaScript stops at the first step.
According to the Vercel and MERJ analysis, the crawlers that fetched JavaScript files still did not execute them. The exceptions were Google's Gemini, which uses Googlebot's rendering infrastructure, and AppleBot. Everyone else saw only what was in the initial HTML.
So a single-page app built with React, Vue or Angular and rendered entirely in the browser can rank reasonably on Google, which does render JavaScript, while being largely invisible to ChatGPT search and Perplexity.
The 2-minute test
You do not need special tools to see what a non-rendering crawler sees:
- Open your page, right-click and choose View page source (not "Inspect", which shows the rendered page).
- Search the source for a sentence from the middle of your page, for example a service description or a price list heading.
- If the sentence is there, crawlers can read it. If you find only
<div id="root"></div>and script tags, they cannot.
From a terminal, curl -s https://yourdomain.com/page | grep "your sentence" does the same job and is easy to repeat for every important page.
The checklist
1. Serve real content in the HTML
This is the fix that matters most. You have three options, from most to least effort to retrofit:
| Approach | What it means | Good fit for |
|---|---|---|
| Server-side rendering (SSR) | The server builds the full HTML on every request | Pages with live data: catalogs, dashboards with public views |
| Static generation / prerendering | HTML is built once at deploy time | Marketing pages, service pages, blogs |
| Static HTML | No framework, plain HTML files | Small sites and landing pages |
Next.js, Nuxt, Astro and SvelteKit all support SSR and static generation. For an existing single-page app, prerendering the public marketing pages at build time is often the quickest win: the app keeps working as before, and crawlers receive finished HTML.
2. Give every page a unique title, description and canonical URL
AI search tools still find candidate pages through search indexes. Google's documentation on AI features is explicit that the usual SEO best practices remain relevant and that there are no extra requirements to appear in AI Overviews. A unique <title>, a meta description that answers the page's main question, and a canonical link remain the foundation.
3. Decide deliberately which AI crawlers you allow
Many sites copied a robots.txt rule that blocks every AI bot. Check what yours says, because the crawlers have different jobs. OpenAI documents that OAI-SearchBot decides whether your site can appear in ChatGPT search, while GPTBot collects content that may be used for model training, and the two are controlled independently. You can allow search visibility while opting out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Google works similarly: the Google-Extended token controls use for Gemini training, not whether you appear in Google Search or AI Overviews. Also check your CDN or firewall: bot-protection settings can block AI crawlers even when robots.txt allows them.
4. Add structured data that describes who you are
Schema.org markup in JSON-LD tells machines what a page is about without guessing. The useful minimum for a business website:
Organization(orLocalBusiness) with your name, logo, address and social profiles, on every pageServiceon service pagesBlogPostingwith author and dates on articlesFAQPagewhere you answer real questionsBreadcrumbListto show how pages relate
5. Write pages that answer questions directly
Generative engines pull passages, not whole pages. Pages that state the answer in the first sentences, use descriptive subheadings, and back claims with numbers and sources are easier to quote. Research on generative engine optimization by Aggarwal and colleagues found that adding citations, quotations and statistics raised a source's visibility in AI answers by up to about 40%.
6. Publish a sitemap and keep dates honest
A sitemap.xml listing every public page, with accurate lastmod dates, helps crawlers find new and updated content quickly. Show a visible "updated" date on articles when you change them substantially, and do not bump dates without real changes.
7. Consider llms.txt, but do not rely on it
llms.txt is a proposed plain-text file that summarises a site for language models. It is cheap to add and can help AI tools that read it, but it is a community proposal rather than a standard that the major search engines have committed to. Treat it as a bonus on top of the items above, never as a replacement.
Frequently asked questions
Does Google have the same problem with JavaScript?
Less so. Googlebot renders JavaScript, although rendering can be delayed and errors can still hide content. Google's Gemini uses the same infrastructure. The gap is mainly with other AI assistants.
Is a WordPress site safe?
Usually yes, because WordPress renders pages on the server. Check page builders and plugins that load content with JavaScript after the page opens, such as some sliders, tabs and pricing tables.
Will blocking GPTBot remove us from ChatGPT?
Not from ChatGPT search. According to OpenAI, blocking GPTBot only opts you out of training. Appearing in ChatGPT search depends on OAI-SearchBot.
How long until changes show up in AI answers?
OpenAI notes that robots.txt changes for its search crawler take about a day to apply. Being cited depends on crawling, indexing and how competitive the topic is, so plan in weeks to months rather than days.
Where to start
Run the two-minute test on your homepage, your main service pages and your contact page. If the content is missing from the page source, fix rendering before anything else; every other optimisation depends on it.
If your site is a single-page app and you are not sure how to move to server rendering without a rebuild, that is exactly the kind of work Kaff Labs does: we audit what crawlers see, then prerender or migrate the pages that bring in customers.
Sources: