Are robots.txt directive names case-sensitive?

Directive names such as User-agent and Disallow are case-insensitive. Path values are case-sensitive because URLs can distinguish uppercase and lowercase characters.

Can I use regular expressions?

No. robots.txt does not support full regular expressions. Major search crawlers support the * wildcard and the $ end anchor for path matching.

Can one group contain multiple user-agents?

Yes. Put consecutive User-agent lines before the group's Allow and Disallow rules. Those crawlers will share the same set of rules.

Can I add more than one Sitemap line?

Yes. Each Sitemap directive should contain one absolute URL, and you may repeat the directive for additional sitemap files.

Is Crawl-delay part of the standard?

No. It is a non-standard extension supported by some crawlers. Google ignores it, so rate controls should be configured through each crawler's supported tools where available.

Syntax

Complete robots.txt syntax

The complete grammar, matching behavior, and compatibility details for writing a valid robots.txt file.

File requirements

Publish it at the site root: https://example.com/robots.txt.
Use plain UTF-8 text. Each record is one field: value line.
It controls only the exact protocol, host, and port where it is served. Subdomains need their own file.
Directive names are case-insensitive; path values are case-sensitive. Spaces around the colon and value are ignored.
A line beginning with #, or text after #, is a comment. A UTF-8 byte-order mark at the beginning is ignored.

# Global rules
User-agent: *
Disallow: /private/
Allow: /private/public/

Sitemap: https://example.com/sitemap.xml

Standard directives

User-agent

User-agent: <crawler-token>

Required at the start of a rule group. It identifies the crawler the following rules apply to. Use * for every crawler. Add consecutive User-agent lines to target several crawlers with the same rules.

Disallow

Disallow: <path-pattern>

Prevents crawling of matching paths for the current group. The value normally begins with /. An empty value means that nothing is blocked.

Allow

Allow: <path-pattern>

Permits a matching path inside a broader Disallow rule. When matching Allow and Disallow patterns are equally specific, Allow wins.

Sitemap

Sitemap: <absolute-URL>

Declares a sitemap using a fully qualified URL. It is independent of user-agent groups, can appear anywhere, and may be repeated for multiple sitemaps.

Groups and precedence

A group begins with one or more consecutive User-agent lines followed by its Allow and Disallow rules. A new User-agent after rules starts a new group.

The crawler selects the group with the most specific matching user-agent token. The wildcard group is used only when no more specific group matches. Groups with the same user-agent token may be combined by crawlers.

For a URL, the matching rule with the longest path wins. If equally specific Allow and Disallow rules both match, Allow wins. File order does not decide the result. With no matching rule, crawling is allowed.

User-agent: Googlebot
User-agent: Bingbot
Disallow: /search/
Allow: /search/help/

User-agent: *
Disallow: /private/

Path pattern syntax

Patterns match from the beginning of the URL path, including parameters and the query string. They do not include the scheme or host.

* matches any sequence of characters, including none.
$ at the end of a pattern anchors the match to the end of the URL.
All other characters are literal. Reserved and non-ASCII characters should use the same percent-encoded UTF-8 form as the URL.

Pattern	Matches
/	Every URL on the host
/admin/	Any path beginning with /admin/
/private$	Exactly /private; $ anchors the match to the end
/*.pdf$	Any URL ending in .pdf; * matches zero or more characters
/search?*	Paths beginning with /search?; the query string is included
/file%20name	A URL containing the percent-encoded space in that path

Non-standard directives

These extensions are not part of the standard Robots Exclusion Protocol. Unsupported crawlers ignore them.

Crawl-delay

Crawl-delay: 10

Supported by Bing and some other crawlers; ignored by Google.

Request-rate

Request-rate: 1/10

Requests one visit per ten seconds; support is limited and crawler-specific.

Host

Host: www.example.com

Legacy preferred-host hint used by some crawlers; ignored by Google and Bing.

Complete example

# Rules for major search crawlers
User-agent: Googlebot
User-agent: Bingbot
Disallow: /account/
Disallow: /search?*
Allow: /account/help/

# Rules for every other crawler
User-agent: *
Disallow: /private/
Disallow: /*.pdf$
Allow: /private/public.pdf$

# Sitemap declarations are global
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/news-sitemap.xml

FAQ

Frequently asked questions

Quick clarification for the syntax details that most often cause confusion.