Free SEO Tool
Robots.txt Generator
Create a robots.txt file to control how search engine crawlers access your site. Set rules per user-agent, add your sitemap, and copy the ready-to-use file.
Robots.txt Settings
Generated robots.txt
# Your robots.txt will appear here
FAQ
Robots.txt FAQ
What is a robots.txt file?
Robots.txt is a text file in your site's root directory that tells search engine crawlers which URLs they can and cannot access. It's a crawl directive, not a security measure.
Where do I upload robots.txt?
Upload it to your website's root directory — the same level as your index.html. The file must be accessible at https://yoursite.com/robots.txt.
Can robots.txt hide pages from search results?
No. Robots.txt blocks crawling, not indexing. If a page is linked from another site, Google can still index it without crawling. Use noindex meta tags to truly prevent indexing.
Does robots.txt affect my SEO?
Indirectly. If you accidentally block important pages, Google can't crawl them and they won't appear in search results. Always test your robots.txt rules before deploying.
Robots.txt is crawl control.
Got your file ready?
Test it with our Robots.txt Tester.
Check if your rules are actually working before you deploy. Our tester checks specific URLs against your robots.txt rules.
Test Your Robots.txt