Crawler groups
User-agent lines define which crawler receives the rules that follow.
robots.txt content
EditorPaste your robots.txt content above (plain text, not a file).
The essentials
What is robots.txt?
robots.txt is a plain text file that gives web crawlers instructions about which parts of a website they can request. It must live at the root of the host, such as example.com/robots.txt, and it can contain separate rules for different crawlers. The robots.txt syntax reference explains each directive and path pattern.
Site owners use it to reduce unnecessary crawling, keep bots away from duplicate or low-value pages, and guide search engines toward important content. It does not hide pages from people and should never be used as a security measure.
Read the complete beginner's guidePath matching
Allow and Disallow patterns are compared against each URL path and query.
Specific rule wins
The longest matching path decides access, with Allow winning equal ties.
Why validate
Small rules can have a big impact
A missing slash, an overly broad wildcard, or a crawler-specific group can change which pages bots reach. Testing real URLs before publishing helps catch accidental blocks and confirms that exceptions work as intended. For live pages, also check HTTP status codes and redirects and check AI crawler access.
Common mistakes to check
- Blocking the entire site with Disallow: /
- Putting full domain names inside path rules
- Expecting wildcard rules to apply to a more specific crawler group
- Using robots.txt to protect private or sensitive pages
FAQ
Frequently asked questions
Everything you need to know before testing crawler access on your website.
Check more than robots.txt
AI Fetch
Check access for ChatGPT, Claude, Gemini, Grok, and Perplexity crawlers.
HTTP Status Checker
Find response errors and follow each URL's redirect chain.
Robots.txt Blog
Read articles about robots.txt, AI crawlers, and technical SEO.