Free tool · no sign-up
Can AI crawlers read your site?
Enter a page. We read its robots.txt, then request it as GPTBot, ClaudeBot, PerplexityBot, Googlebot and four more, and show what each one was actually sent.
What the check does
- Reads robots.txt and works out, for each crawler, whether the rules allow the page you entered, including groups that name the crawler.
- Requests the page as each crawler, with the user agent its operator publishes, and records the status code and how much text came back.
- Requests it as a normal browser, so a crawler handed a refusal, a challenge or an empty shell stands out.
One limit: the requests come from our servers, not from OpenAI’s or Google’s. A firewall that verifies crawler addresses may refuse our imitation while letting the real crawler in, and the result says so when that is likely.
Questions
- Why check the server as well as robots.txt?
- robots.txt is a request. A CDN, firewall or bot-management rule decides what a crawler actually receives, before your server sees the request, and it can refuse a crawler your robots.txt allows. Only a real request as that crawler shows the difference.
- Which crawlers matter for AI answers?
- OAI-SearchBot and ChatGPT-User decide whether ChatGPT can read and cite you; GPTBot collects training data. PerplexityBot and Perplexity-User do the same for Perplexity. Googlebot covers Google Search, AI Overviews and AI Mode, and the Google-Extended token governs Gemini's use of your pages.
- What does a near-empty page mean?
- The crawler got under 400 characters of text while a browser got the full page. Some sites render content only with JavaScript, which AI crawlers rarely run, so the crawler sees a shell.
- Is the result stored?
- No. The check runs when you press the button and the result lives only on your screen.