Testing with AI crawlers...
Crawler Results
About These Crawlers
GPTBot (OpenAI)
OpenAI's web crawler that collects content to train and improve AI models like ChatGPT. Respects robots.txt and filters out paywalled content.
View Documentation →ClaudeBot (Anthropic)
Anthropic's crawler for AI model training and web search. Respects crawl-delay settings and anti-circumvention technology.
View Documentation →Googlebot (Google)
Google's primary crawler running on thousands of machines simultaneously. Supports HTTP/2, compression, and uses caching for efficiency.
View Documentation →BingBot (Microsoft)
Microsoft's web crawler for Bing search. Supports IndexNow protocol for instant crawling and offers crawl control tools.
View Documentation →