Crawling, Scraping, and Site Controls
Crawling typically means automatically discovering and fetching URLs at scale. Scraping (web scraping) emphasizes extracting structured data from pages or APIs — text, tables, prices, and other fields. In practice the terms are often used interchangeably; many programs both follow links and extract data fields.
Crawling is not the same as indexing. A search engine may fetch a URL without ever including it in search results, or display a URL without a recent crawl. The robots.txt file (governed by RFC 9309) asks cooperative crawlers which paths not to fetch, while noindex meta tags and X-Robots-Tag headers steer indexing behavior. A common misconfiguration: using Disallow to block a page entirely prevents crawlers from ever seeing a noindex directive on that page.
Google-Extended is a separate robots.txt user-agent token that controls whether content may be used for certain Gemini-related training and grounding applications. Google's documentation states it does not function as a standalone switch for traditional search rankings — HTTP requests may still use familiar Google user agents. Configure training, search indexing, and AI retrieval permissions deliberately as separate decisions.
Search Engine Crawlers
Search engine crawlers are the automated clients that fetch pages so engines can build and refresh their indexes. The major search crawlers include Googlebot (Google Search and Gemini AI), Bingbot (Microsoft Bing), YandexBot (Yandex), and Baiduspider (Baidu). These crawlers have been optimized over decades to efficiently discover and render content at web scale.
Googlebot is the most active, generating approximately 4.5 billion requests monthly on the Vercel network alone. It uses the Chrome rendering engine to process JavaScript, fully rendering modern web applications including SPAs built with React, Vue, and Angular. Bingbot has comparable JavaScript rendering capabilities. These crawlers operate from multiple geographic locations to ensure global coverage and load balancing.
Despite full JavaScript rendering support, excessive or slow-loading JavaScript still affects crawl efficiency. Website owners should use server-side rendering (SSR) or static site generation (SSG) to ensure critical content — article bodies, product information, metadata — is present in the initial HTML response rather than relying entirely on client-side JavaScript execution.
JavaScript Rendering Capability
Modern search crawlers (Googlebot, Bingbot) have full JavaScript rendering capability, executing scripts and rendering complete page content. Googlebot uses the Chrome rendering engine, handling CSS, Ajax requests, WebSocket connections, and other modern web technologies. This means SPAs can be correctly crawled and indexed — but JavaScript rendering consumes additional resources, and content dependent on slow or excessive JavaScript may not be fully processed within crawl budget constraints.
AI Crawlers
AI crawlers are operated by AI companies (OpenAI, Anthropic, Perplexity, ByteDance) to collect web data for training large language models and powering real-time retrieval features. According to Vercel and MERJ research, AI crawler traffic is substantial: OpenAI's GPTBot generated 569 million requests on the Vercel network in a recent month, while Anthropic's ClaudeBot generated 370 million — combined, approximately 20% of Googlebot's 4.5 billion requests in the same period.
Types of AI Crawlers
AI crawlers serve two primary purposes. Model training crawlers (GPTBot, ClaudeBot) continuously collect diverse web content — HTML, images, JavaScript files — to train and optimize language models. Real-time retrieval crawlers (ChatGPT-User, OAI-SearchBot) fetch pages during user queries to provide current information and citation links, typically referencing existing search engine indexes to identify relevant pages.
Major AI Crawlers and User Agents
OpenAI operates three distinct crawler identities: GPTBot for model training, ChatGPT-User for user-triggered real-time retrieval, and OAI-SearchBot for building ChatGPT Search indexes. Anthropic documents separate bots: ClaudeBot for training, Claude-SearchBot for search quality and indexing work, and Claude-User for user-directed retrieval. Other notable AI crawlers include PerplexityBot, Bytespider (ByteDance), and Amazonbot. Each can be targeted independently in robots.txt.
Most AI crawlers operate from US data centers — ChatGPT from Iowa and Arizona, Claude from Ohio — in contrast to search crawlers that distribute crawling across multiple global locations.
AI Crawler Behavior Characteristics
AI crawler behavior differs significantly from search crawlers in several dimensions. On JavaScript rendering: most AI crawlers (GPTBot, ClaudeBot, PerplexityBot) do not execute JavaScript, only reading initial HTML. Only Google's Gemini (using Googlebot infrastructure) and AppleBot have full JavaScript rendering. CSR-dependent web applications may be invisible to AI training crawlers.
On content type priorities: ChatGPT prioritizes HTML (57.70% of requests), while Claude focuses heavily on images (35.17%). Both spend significant time crawling JavaScript files (ChatGPT: 11.50%, Claude: 23.84%) despite not executing them — likely because AI models learn from JavaScript code as text data.
On crawling efficiency: AI crawlers show notably higher error rates than search crawlers. ChatGPT has 34.82% of requests returning 404, Claude 34.16%, and ChatGPT follows redirects on 14.36% of requests. By comparison, Googlebot has only 8.22% 404s and 1.49% redirects — reflecting decades of optimization in URL discovery and validation that AI crawlers have not yet matched.
Verifying Real Crawler Identity
User-Agent strings are trivially spoofed and should not be the sole basis for access decisions. Google and Bing publish verification procedures — typically reverse DNS lookups or IP allowlists. Yandex and Baidu similarly document reverse DNS patterns for their networks. For AI crawlers, some providers offer cryptographic verification: OpenAI documents HTTP Message Signatures (RFC 9421) with a Signature-Agent header for ChatGPT agent traffic, providing a provable identity mechanism beyond UA string matching. Always consult each provider's current documentation before blocking based on user agent alone.
Browser Agents, Third-Party, and Non-Search Bots
Some AI products use browser automation flows that do not appear as traditional bot user-agent strings. OpenAI's ChatGPT agent can be validated through cryptographic HTTP Message Signatures (RFC 9421) and a Signature-Agent header — a fundamentally different control surface than listing GPTBot in robots.txt. Similarly, do not equate ChatGPT-User (user-triggered fetches) with search index bots; OpenAI's documentation notes that robots rules for fully automatic crawling may not apply to certain user-initiated actions.
Beyond search and AI vendors, your origin server commonly encounters SEO platform crawlers (link indexes, site audits), social link preview fetchers (pulling Open Graph tags for messaging and social apps), RSS and feed readers, uptime monitors, and archival projects like Common Crawl (CCBot). These bots may be well-behaved and rate-limited — or noisy if misconfigured. They are not search crawlers but still consume server resources and bandwidth, and should be identified and managed separately in traffic analysis.
How to Manage Crawler Access
Website owners need to decide on crawler access policies based on brand goals and risk assessment. For e-commerce sites, allowing major AI crawlers can be beneficial — contributing brand narrative and enabling product visibility through real-time retrieval. For content publishers, more nuanced strategies may be needed to balance visibility against the risk of AI search summarization reducing organic traffic.
Strategies to Enable Crawler Access
Prioritize server-side rendering for critical content. Since most AI training crawlers do not execute JavaScript, important content — articles, product information, metadata, navigation structure — must be present in the initial server-rendered HTML. Use SSR, ISR, or SSG to ensure all crawlers can access your content. Client-side rendering remains appropriate for non-essential dynamic elements like view counters, interactive UI enhancements, live chat widgets, and social feeds.
Maintain clean URL management. AI crawlers' high 404 rates highlight the importance of correct redirects, updated sitemaps, and consistent URL patterns. Ensure all important pages resolve correctly without redirect chains or broken links. Use sitemaps and the IndexNow protocol to proactively notify search engines of new content, encouraging crawlers to focus on your priority pages.
Most strategies marketed as GEO (Generative Engine Optimization) are fundamentally SEO fundamentals applied to AI crawler visibility. On a foundation of user-friendly, search-engine-friendly content, ensure the same content is accessible to AI crawlers and LLM web search APIs. Strong SEO foundations — proper HTML structure, fast loading, clear metadata — benefit both traditional search and AI-driven discovery.
Strategies to Block Crawler Access
Use robots.txt to control access. Specify user-agent tokens for each crawler you wish to restrict, setting Disallow rules for sensitive or unnecessary paths. Check each company's current documentation for the correct user-agent strings — Applebot and OpenAI crawlers are documented publicly.
Deploy firewall rules. Vercel's WAF provides AI bot firewall rules for one-click blocking. Other CDNs and hosting platforms offer similar bot management features. For more granular control, combine robots.txt with edge-level WAF policies.
Consider the trade-offs. Indiscriminately blocking all AI crawlers may cause you to miss consumers searching for your products on non-Google platforms. AI model knowledge is bounded by training data — if brands block all AI crawlers, these models will learn about brands from third-party sites, reviews, and competitors. The only way to maintain brand narrative control in AI search is to contribute to what models know about your brand.
Bot Traffic Management
According to Imperva's (Thales subsidiary) 2025 Bad Bot Report, automated bot traffic exceeded human-generated traffic for the first time in 2024, accounting for 51% of global internet traffic. Malicious bots alone represented 37% of total internet traffic, up from 32% in 2023 — the sixth consecutive year of growth. AI tool proliferation has lowered barriers for attackers, enabling large-scale creation of increasingly sophisticated bots that mimic human behavior and evade traditional security measures.
This shift has profound operational implications. Over half of internet traffic now comes from automated programs rather than humans, potentially distorting analytics, inflating ad performance metrics, and disrupting recommendation algorithms. The travel industry is particularly affected: malicious bots accounted for 41% of travel traffic in 2024, making it the most attacked sector.
Bot traffic divides into two categories. Good bots — search engine crawlers, AI crawlers, monitoring tools — serve legitimate purposes and should generally be allowed. Bad bots — content scrapers, spam bots, credential stuffers, DDoS agents — cause data leaks, server overload, and content theft. Identification methods include analyzing user behavior patterns (bots show repetitive, predictable behavior), checking user-agent consistency, monitoring IP geography and reputation, and comparing session duration and page-view patterns against human baselines.
Management strategies include deploying WAF rules to filter malicious bots, setting rate limits to prevent excessive access, using CAPTCHA challenges for suspicious traffic, configuring robots.txt for cooperative crawlers, and regularly auditing server logs for anomalous patterns.
However, not all bots claiming compliance actually respect robots.txt. WIRED's June 2024 investigation found that Perplexity used an undisclosed IP address to bypass robots.txt restrictions, secretly crawling content from Condé Nast properties at least 822 times over three months. Perplexity's chatbot also demonstrated hallucination problems — in tests, it fabricated content rather than accurately summarizing articles. Similarly, ByteDance's Bytespider, released April 2024, crawls approximately 25 times faster than GPTBot and 3,000 times faster than ClaudeBot, and has been documented ignoring robots.txt directives. These cases underscore that robots.txt alone is insufficient for access control — it must be combined with WAF policies, rate limiting, and server-side verification.
Conclusion
Web crawlers are foundational infrastructure for both search engines and AI systems, playing critical roles in content discovery and indexing. Search crawlers (Googlebot, Bingbot) are highly optimized with full JavaScript rendering, efficiently building the indexes that power traditional search. AI crawlers (GPTBot, ClaudeBot) represent a newer but rapidly growing category, serving model training and real-time retrieval functions with different technical characteristics — notably, limited JavaScript execution and higher error rates.
Site owners should map the four distinct purposes — crawl, index, train, and retrieve — and configure robots.txt tokens, rendering strategy, and security controls for each category separately. For sites seeking visibility, SSR or SSG for critical HTML remains essential, especially given that many AI training crawlers do not execute JavaScript. For sites seeking restrictions, combine robots.txt directives with edge-level WAF policies and, where appropriate, authentication — never rely on user-agent string blocking alone.
For deeper guidance on indexing, sitemaps, robots rules, and internal linking, see the companion articles on website indexing, XML sitemaps, robots.txt configuration, and internal link strategy.
References
- RFC 9309: Robots Exclusion Protocol (IETF · 2022) — Formal specification for the robots.txt protocol governing crawler access.
- Google's Common Crawlers (including Google-Extended) (Google for Developers · 2026) — Official documentation of Google's crawler user agents and Google-Extended token behavior.
- OpenAI Crawler Documentation (OpenAI · 2026) — OpenAI's official documentation for GPTBot, ChatGPT-User, OAI-SearchBot, and crawler configuration.
- 2025 Bad Bot Report (Imperva (Thales) · 2025) — Annual report on global bot traffic trends, including the finding that bots exceeded human traffic for the first time in 2024.