How to Check If AI Can Read Your Website: 3 Free Tools

What does it mean for AI to “read” a website?

The phrase sounds simple, but it covers three separate layers.

A website can pass one layer and fail another. For example, a crawler may reach the server successfully but receive a nearly empty HTML shell. Another site may expose all its content but block the crawler at the edge.

That is why one test is not enough.

Why Google rankings do not prove AI readability

Google can process JavaScript and render pages before indexing them. Many AI crawlers and agent tools depend more heavily on the HTML returned by the server.

This creates a common problem on React and other client-rendered websites.

A browser may display a complete page after JavaScript loads, while the original response contains little more than:

```html
<div id="root"></div>
```

The same issue can affect product descriptions loaded after page load, FAQs stored in a client-side JSON object, and text added through an API request after the initial HTML arrives.

In that situation, Google may understand the rendered page while another crawler sees very little useful content. Good rankings therefore do not prove that an AI assistant can retrieve the same information.

The most important question is not simply whether the page works in a browser. It is whether the page delivers useful, structured content to a crawler before client-side rendering begins.

Tool 1: Check agent readiness with Is Agentic

Is Agentic is a free website scanner from Vercel that evaluates how easily AI agents can discover, retrieve, understand and use a public website.

Enter your domain and the tool returns:

• an overall readiness score;
• essential and recommended checks;
• evidence behind each result;
• actionable recommendations for failed or partial checks.

The essential checks focus on fundamentals such as server-rendered content, HTTP behaviour, document structure, usable controls and recoverable errors. Recommended checks appear when the scanner detects relevant capabilities such as an API, OAuth, GraphQL, commerce functionality, a developer portal or an MCP server.

This makes Is Agentic a useful first step because it does more than show a score. It explains what caused the score and what should be fixed.

How to run the check

Open is-agentic.com, enter your domain and review the essential failures first.

You can also run the scanner from a terminal:

```bash
npx is-agentic yourdomain.com
npx is-agentic yourdomain.com --json
```

How to interpret the score

Treat the score as a prioritized technical review, not a permanent grade or a guarantee that every AI system can use the site.

Pay more attention to the failed checks than the total number. A moderate score caused by optional features is less urgent than a high score hiding a critical problem such as missing server-rendered content.

Also remember that reports may be cached for a domain. Use the rescan option after making changes so that you are reviewing a current result.

What to review first in the report

Start with these findings:

1. Content without JavaScript — Is useful page content present in the original response?
2. Bot access — Are known AI user agents being blocked by security rules?
3. Redirect hygiene — Are redirects handled through proper HTTP responses rather than JavaScript-only logic?
4. Error recovery — Do missing pages return a real status code and provide a useful next step?
5. Document structure — Does the page use a clear heading hierarchy and descriptive controls?

Tool 2: See real crawler activity with Cloudflare AI Crawl Control

Is Agentic shows whether your website is technically readable. Cloudflare AI Crawl Control shows which AI services are actually reaching it.

If your website uses Cloudflare, the dashboard can help you:

• monitor AI crawler activity and request patterns;
• see which services are requesting your content;
• allow or block individual crawlers;
• review robots.txt compliance;
• identify whether security rules are stopping expected traffic.

This is important because a robots.txt file describes your preference, but it is not the same as an enforced firewall rule. Your WAF, bot-management settings and Cloudflare crawler controls can affect what reaches the origin.

What to check in the dashboard

Focus on three questions.

1. Are the crawlers appearing at all?

Look for the services that matter to your visibility goals. If a search or retrieval crawler never appears, investigate whether it is being blocked before it can access your pages.

2. Are requests being allowed or blocked?

Review Cloudflare rules, WAF policies and bot-protection settings. A broad anti-scraping rule may block useful AI traffic along with unwanted crawlers.

3. Has crawler activity changed over time?

A crawler that previously visited the site and then stopped may indicate a new rule, deployment issue, server error or access-policy change.

Training bots, search bots and user-initiated fetchers are different

Do not treat every AI user agent as the same type of crawler. The purpose determines the visibility impact of blocking it.

• Training crawler — GPTBot, ClaudeBot, Google-Extended. Collect content that may be used to improve or train models. Usually affects model-training access rather than live search eligibility.
• Search or retrieval crawler — OAI-SearchBot, PerplexityBot. Discover and retrieve pages for AI search experiences. Blocking can prevent the site from appearing in relevant AI search results.
• User-initiated fetcher — ChatGPT-User, Perplexity-User. Visit a page in response to a specific user action or question. Blocking can stop the requested page from being accessed during that interaction.
• Traditional search crawler — Googlebot, Bingbot. Build conventional search indexes. Blocking can damage standard search discovery and indexing.

This distinction lets you make a deliberate policy. A company may choose not to allow training access while still allowing search and user-requested retrieval.

Tool 3: Check individual pages with a browser extension

The first two tools operate mainly at the domain level. A browser extension is faster when you need to answer a page-specific question:

Does this page contain useful content before JavaScript runs?

Useful free options include:

• BotLens — scores a page for AI readability and compares the initial HTML with the rendered DOM, making it useful for detecting JavaScript-only content.
• AI Bot Access Checker — checks the current website’s robots.txt rules against known AI bots, search crawlers, SEO tools and social-preview bots.
• LLMs.txt Checker — checks whether the website publishes llms.txt or llms-full.txt at the root.

Do not test only the homepage. Check the pages most likely to answer a buyer’s question:

• product or service pages;
• pricing pages;
• documentation;
• comparison pages;
• case studies;
• important blog articles;
• About and Contact pages.

A website can have a readable homepage and still hide its most valuable commercial content behind client-side rendering.

The 10-minute AI readability audit

Run the checks in this order. Each step answers a question the previous step cannot.

For the final step, open:

```text
https://yourdomain.com/robots.txt
```

Review any Disallow rules that apply to AI user agents. Also look for duplicate or conflicting groups created by different plugins, managed rules or manual edits.

Remember that robots.txt communicates crawl preferences. It does not secure private information, and different crawlers may handle its rules differently.

What should you fix first?

Not every warning has the same impact. Prioritize the issues that prevent access or remove useful content.

1. Serve primary content in the initial HTML

Your page title, main explanation, product details, pricing context and key links should not depend entirely on client-side JavaScript.

For JavaScript applications, use server-side rendering, static generation or another method that returns meaningful content in the original HTML response.

2. Review access for search and retrieval crawlers

Decide deliberately which services you want to allow. Do not use one broad bot rule without understanding which training, search and user-initiated agents it affects.

3. Check WAF and bot-protection rules

A correct robots.txt file will not help when a security rule blocks the request first. Review Cloudflare, CDN, hosting and application-level protections.

4. Remove conflicting robots.txt rules

Duplicate groups for the same user agent can create confusing or inconsistent behaviour. Keep the policy clear and maintain it in one place where possible.

5. Use real HTTP redirects and error responses

Return proper 301 or 302 responses for redirects. Return a real 404 status for missing pages and include helpful links so an agent can recover rather than reaching a dead end.

6. Improve structure and entity information

Use one clear H1, logical H2 and H3 sections, descriptive link text and understandable form labels.

Complete your Organization schema with accurate details such as the company name, category, address and contact information. Publish useful About and Contact pages so both people and systems can resolve who the business is.

What these tools can and cannot tell you

These checks tell you whether AI systems can access and interpret your website.

They do not tell you whether an assistant will recommend your brand when a buyer asks a commercial question.

Recommendation visibility depends on additional signals, including:

• whether your pages clearly answer the question;
• whether independent sources mention and describe the brand;
• whether the information is consistent across the web;
• whether the system considers the source relevant and trustworthy for that query.

Reachability still comes first. External mentions cannot help a retrieval system use your website when the crawler receives an empty page or is blocked before it reaches the content.

Run the three checks first. If the site passes, your next problem is likely content coverage and third-party evidence. If it fails, you have a technical issue worth fixing before doing more AI visibility work.

Useful implementation resources

Tools

• Is Agentic
• Cloudflare AI Crawl Control
• Cloudflare AI Crawl Control documentation
• BotLens
• AI Bot Access Checker
• LLMs.txt Checker

Crawler documentation and standards

• OpenAI crawler documentation
• Perplexity crawler documentation
• Google crawler documentation
• RFC 9309: robots.txt specification
• llms.txt proposal
• Schema.org Organization

See what AI says about your brand

Technical accessibility is only the first layer of AI visibility.

Xtrusio audits how brands appear across ChatGPT, Gemini, Claude and Perplexity—including which competitors are mentioned, which sources are cited and where the evidence gaps exist.

Built by the team at imaPRO for brands competing across Bahrain, the wider Gulf, India and international markets.

Continue reading

• How AI Models Choose Which Brands to Recommend: The Signals That Matter in 2026
• What Are Semrush AI Visibility Products? Definition, Workflow and Real Use Cases