AI generator
Describe the rules. Get the file.
Tell the assistant what crawlers should access, block, or ignore. Refine the result in the same conversation. Not sure which rules you need? Read the robots.txt guide first.
robots.txt assistant
Ready to design your crawl rules
What should crawlers do?
Include the paths to allow or block, the crawler names if relevant, and your sitemap URL if you have one.
FAQ
Frequently asked questions
How to get a useful file from the assistant and safely put it into use.
Before you publish your file
Robots.txt Tester
Check URLs against your rules and find the exact matching line.
Robots.txt Syntax
Look up directives, wildcards, and complete file examples.
AI Fetch
Check access for ChatGPT, Claude, Gemini, Grok, and Perplexity crawlers.