Free robots.txt Generator
Choose your rules and generate a valid robots.txt file for your website — control which crawlers can access which paths, and point search engines to your sitemap. It runs instantly in your browser session, no sign-up required.
robots.txt Generator
Generate a valid robots.txt file for your website.
About this robots.txt generator
A robots.txt file sits at the root of your website and tells search engine crawlers which parts of your site they may or may not access. It is one of the first files crawlers look for, so getting it right helps search engines index the pages you want and skip the ones you do not.
This generator builds a correctly formatted robots.txt from simple options: allow all crawlers, block all crawlers, or set custom allow and disallow rules for specific paths. You can target a specific user-agent, add an optional crawl-delay, and include your sitemap URL, which is a recommended best practice.
Copy the generated file and save it as robots.txt in the root directory of your site (so it is reachable at yourdomain.com/robots.txt). Remember that robots.txt guides cooperating crawlers but does not enforce security — do not rely on it to hide sensitive pages.
Frequently asked questions
How does the robots.txt generator work?
Pick a rule (allow all, block all, or custom), add any paths and your sitemap URL, and click Generate. Copy the result into a robots.txt file at your site root.
Where do I put the robots.txt file?
In the root directory of your website, so it is accessible at yourdomain.com/robots.txt.
What is the difference between allow and disallow?
Disallow tells crawlers not to access a path; Allow explicitly permits a path, useful for re-allowing something inside a disallowed folder.
Should I include my sitemap?
Yes. Adding your sitemap URL is a best practice that helps search engines discover all your pages.
Does robots.txt keep pages private?
No. It guides cooperating crawlers but does not enforce access control. Do not use it to protect sensitive content — use proper authentication for that.