Question 1

What should I include in my request?

Accepted Answer

Describe the paths to allow or block, the crawlers those rules should apply to, and any sitemap URL you want included. You can also explain your site type for useful defaults.

Question 2

Can I refine the generated file?

Accepted Answer

Yes. Continue the same conversation and ask for changes such as adding a crawler, allowing a specific subfolder, or removing a rule.

Question 3

Should I test the generated robots.txt?

Accepted Answer

Yes. Review the file and test important public and private paths in the Validator before publishing it to your website.

Question 4

Does the generator protect private pages?

Accepted Answer

No. A generated robots.txt only gives voluntary crawling instructions. Private pages still need authentication and server-side access controls.

Describe the rules. Get the file.

robots.txt assistant

What should crawlers do?

Frequently asked questions