Guide
How robots.txt works
A robots.txt file tells crawlers which parts of your site they may fetch. It is a plain text file served at the root of your domain. A single misplaced line can hide an entire section from search engines.
1. User-agent groups
Rules are grouped under one or more User-agent lines. A crawler looks for the group that matches its name most specifically. If a group for Googlebot exists, Googlebot obeys only that group and ignores the User-agent: * group entirely. If no specific group matches, the wildcard group applies.
2. Allow and Disallow rules
Inside a group, each rule is a path pattern. Disallow: /admin/ blocks every URL whose path starts with /admin/. An empty Disallow: means nothing is blocked. Rules are evaluated against the URL path including the query string.
3. Wildcards
* matches any sequence of characters, and $ anchors the pattern to the end of the URL. For example, Disallow: /*.pdf$ blocks every URL ending in .pdf.
4. Longest match wins
When several rules match one URL, the longest pattern wins, regardless of order. If an Allow and a Disallow pattern have the same length, the Allow wins. When no rule matches, the URL is allowed by default.
Paste
Drop your robots.txt content and the URLs you care about.
Test
Pick a crawler and run the test against Google's matching rules.
Verify
See the exact rule that allows or blocks each URL, highlighted in place.
Ready to check your rules?
Paste your robots.txt and find out exactly what every crawler can reach.
Open the testerFAQ
Frequently asked questions
Practical answers about publishing and maintaining your robots.txt file.