Skip to content
Back to glossary

Glossary

AI Crawlers

AI crawlers are bots like GPTBot, PerplexityBot and ClaudeBot that index web content for AI search engines, unlike traditional search engine crawlers.

AIteknikindexering

AI crawlers are automated bots that index web content specifically for AI search engines and large language models. The most important AI crawlers include GPTBot (OpenAI/ChatGPT), PerplexityBot (Perplexity), ClaudeBot (Anthropic), and Google-Extended (Google's AI services). These crawlers differ from traditional search engine crawlers like Googlebot in several ways.

How do AI crawlers differ from traditional crawlers?

Traditional crawlers like Googlebot index web pages to build a search index. AI crawlers collect data used to train or inform large language models (LLM). While Googlebot visits pages to assess ranking signals, AI crawlers collect content that can be used as source material for AI-generated answers.

Which AI crawlers exist?

  • GPTBot: OpenAI's crawler that collects data for ChatGPT.
  • PerplexityBot: Perplexity's crawler that indexes for their AI search engine.
  • ClaudeBot: Anthropic's crawler.
  • Google-Extended: Google's crawler for AI services beyond traditional search.

Should you block or allow AI crawlers?

If you want to appear in AI-generated answers, you should allow AI crawlers to index your content. You can control this via robots.txt. Companies actively working with GEO should ensure AI crawlers have access to all relevant content.