Question 1

What does this AI Fetch check actually test?

Accepted Answer

For every URL we download the site robots.txt, read the page response status, and look for X-Robots-Tag headers and robots meta tags. Then we apply the robots.txt rules for each AI crawler user agent, exactly the way a crawler would.

Question 2

Does a green result guarantee my page ends up in an AI answer?

Accepted Answer

No. It means nothing in robots.txt or in your page directives stops that crawler from fetching the URL. Whether a model uses the content is up to the provider.

Question 3

Why is a page allowed in robots.txt but still flagged?

Accepted Answer

A page can be crawlable and still carry a noindex or noai directive in an X-Robots-Tag header or a robots meta tag, or return an error status. We flag those separately because robots.txt alone will not reveal them.

Question 4

How do I let AI crawlers in again?

Accepted Answer

Remove or narrow the Disallow rule in the group that names the crawler, or add a more specific Allow rule for the path inside the same group. The tips shown next to each blocked result give you the exact lines.

Question 5

Should I allow every AI crawler?

Accepted Answer

It is a business choice. Many sites allow answer and search crawlers such as OAI-SearchBot and PerplexityBot for visibility, while blocking pure training crawlers such as GPTBot, Google-Extended, and xAI-Bot.

AI Fetch

Frequently asked questions

Investigate crawler access