robots.txt: Definition, Syntax & Best Practices

1. Optimize Crawl Budget {#}

Search engines allocate a limited crawl budget to each site — the number of pages they'll crawl in a given timeframe. For larger sites (thousands of pages or more), robots.txt lets you direct that budget toward your most important content by blocking low-value URLs:

2. Protect Sensitive Resources {#}

While robots.txt is not a security mechanism, it prevents well-behaved crawlers from wasting resources on areas that should not appear in search results: admin panels, API endpoints, internal scripts, and private directories.

3. Manage AI Crawlers {#}

The rise of large language models has introduced a new category of bots. Some crawl to build training datasets (GPTBot, ClaudeBot, Bytespider); others crawl to provide cited search results (OAI-SearchBot, PerplexityBot, Claude-User). robots.txt is currently the primary mechanism for declaring your preferences to both categories.

A growing number of sites distinguish between these two groups: allowing AI search/citation bots (which bring visibility and referral traffic) while disallowing AI training bots (which consume bandwidth without attribution).

1. Basic Directives {#}

2. Pattern Matching {#}

3. Priority Rules {#}

When multiple rules could apply to the same URL: more specific paths take precedence over general ones. Allow: /shop/shoes/` overrides Disallow: /shop/`. For the same path, the first matching rule in the file wins — so put specific Allow rules before broad Disallow rules.

1. Use a Structured, Commented Format {#}

Group rules by purpose and add comments explaining why each section exists. A future maintainer (including you, six months later) should understand the reasoning without guessing.

# === Crawl budget: block low-value paths ===
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
# === Allow blog content indexing ===
Allow: /blog/*.html$
# === Sitemap reference ===
Sitemap: https://example.com/sitemap.xml

2. Handle Multiple Subdomains Explicitly {#}

Each subdomain needs its own robots.txt at its root. For consistency across subdomains, some teams use a redirect to a central file: Redirect 301 /robots.txt https://cdn.example.com/robots.txt` Note: this only works if the central file uses path-relative rules that apply correctly to all subdomains.

3. Separate Rules by Crawler {#}

Different bots serve different purposes. A modern robots.txt typically has at least two User-agent blocks: one for traditional search engines (broad Allow) and one for AI training bots (broad Disallow). See the strategy guide below for decision frameworks.

1. Blocking Critical Resources {#}

The mistake: Disallow: /css/` or Disallow: /js/` prevents crawlers from fetching stylesheets and scripts. Google uses these to understand page layout and mobile-friendliness — blocking them can hurt rankings.

The fix: Never block asset directories by default. If you need to hide specific resources, use more granular path patterns.

2. Using Absolute Urls {#}

The mistake: Disallow: https://example.com/private/` Why it fails: robots.txt directives use path-relative notation only. Absolute URLs are silently ignored by most parsers.

The fix: Always use relative paths: Disallow: /private/`.

3. Ignoring Case Sensitivity {#}

URL paths are case-sensitive. Disallow: /Admin/` does not block /admin/`. Always match the actual URL casing used by your server.

4. Forgetting to Update after Site Changes {#}

A robots.txt rule that blocked /old-blog/` two years ago may now be blocking your new blog if you repurposed that path. Audit your robots.txt whenever you restructure URLs or launch new site sections.

1. Dynamic Parameter Control {#}

For CMS-driven sites that generate URLs with query parameters (WordPress, Shopify, etc.), you can block parameter-heavy pages while allowing specific tracked parameters:

User-agent: 
Disallow: /?
Allow: /?utm_source
Allow: /?utm_medium
Allow: /*?ref

This allows marketing attribution links while blocking other dynamically-generated duplicate pages.

2. Crawler-Specific Rules {#}

Different search engines have different crawlers for different content types. You can target rules to specific ones:

# Allow news crawler to access press releases only
User-agent: Googlebot-News
Allow: /press-releases/
Disallow: /
# Block Image Crawler from Heavy Asset Directories
User-agent: Googlebot-Image
Disallow: /assets/originals/

3. Crawl-Delay Directive {#}

The Crawl-delay` directive requests a minimum delay (in seconds) between successive requests. Note: Google does not support this directive; it is primarily honored by Bing and Yandex. Crawl-delay: 10`

1. Google Search Console robots.txt Tester {#}

Navigate to Settings → robots.txt in Google Search Console. The built-in tester highlights syntax errors, flags conflicting rules, and lets you test specific URLs against your rules before making them live. This is the single most important validation step.

2. Server Log Analysis {#}

Monitor your access logs to verify which crawlers are hitting your site and whether they're respecting your robots.txt rules. A typical Googlebot entry looks like:

66.249.66.1 - - [15/Jul/2024:12:34:56 +0000] "GET /blog/ HTTP/1.1" 200 4321 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

If you see a blocked user-agent accessing disallowed paths, verify your robots.txt syntax and check that your web server is serving it correctly (Content-Type: text/plain, HTTP 200).

3. Third-Party Crawl Tools {#}

Tools like Screaming Frog and Sitebulb can simulate how search engines interpret your robots.txt and identify pages that are accidentally blocked or exposed.

1. Google — Clean and Self-Referential {#}

Google's own robots.txt is famously minimal and well-commented. One of their most shared easter eggs references the Terminator films:

User-agent: T-800
User-agent: T-1000
Disallow: /+LarryPage
Disallow: /+SergeyBrin

Beyond the humor, the real lesson is in the file's structure: clear comments explaining each section, grouped rules by crawler type, and a disciplined approach to only blocking what needs blocking.

2. Nike — Brand Identity in Configuration {#}

Nike extends its "Just Do It" philosophy into robots.txt with a playful comment. The more practical takeaway: their file is structured to protect product catalog pages while allowing marketing and brand content to be fully crawlable — a pattern relevant to any e-commerce site.

3. Wikipedia — Minimalist and Open {#}

Wikipedia's robots.txt takes the opposite approach from most commercial sites: it blocks almost nothing, relying on its internal linking structure and canonical tags to manage crawl efficiency. This demonstrates that for sites with naturally strong architecture, a minimal robots.txt is often the best choice.

Small Sites (< 50 Pages): Keep It Minimal {#}

If your site has fewer than 50 pages with strong internal linking, you likely do not need complex robots.txt rules. A minimal file with just a sitemap reference is often sufficient:

Sitemap: https://yoursite.com/sitemap.xml

Add Disallow rules only if you have specific pages that should not appear in search results (admin panels, thank-you pages, internal tools).

Content Sites and Blogs: Focus on AI Bot Policy {#}

For content-driven sites, the biggest robots.txt decision in 2024–2025 is your AI crawler policy. A common approach:

Whether to also disallow Google-Extended (which controls use in Google's AI products) depends on your content strategy — if you rely on Google Discover or AI Overviews for traffic, blocking it may be counterproductive.

E-Commerce Sites: Prioritize Crawl Budget {#}

For product catalogs with hundreds or thousands of SKUs, crawl budget is your primary concern. Block faceted navigation, internal search, sorting parameters, and session IDs. Allow product pages, category pages, and your blog. Review your parameter list quarterly against your analytics to ensure you're not accidentally blocking newly important sections.

SaaS and Web Apps: Protect Gated Resources {#}

SaaS sites typically have a public-facing marketing site and a gated application. Your robots.txt should block the app's internal routes (dashboards, settings, API docs) while keeping marketing pages fully crawlable. If your app and marketing site share a domain, this separation in robots.txt is essential.