Robots.txt Generator
Write a robots.txt that does what you intend — and test any URL against it the way Google does.
- Your data never leaves your device
- 100% free
- No sign-up
- Works offline after first load
How to create a robots.txt file
- Pick a template or fetch your current robots.txt.
- Add rules and your sitemap, and fix any warnings.
- Test a few URLs, then download robots.txt and upload it to your site root.
About Robots.txt Generator
Create, edit and test robots.txt.
It works in your browser, so it is instant and private, and your settings are remembered on this device.
Good for
- New websites and redesigns
- Blocking admin, cart and search pages
- Controlling AI crawlers
Tip: Never block CSS and JavaScript files — Google needs them to render your pages.
Frequently asked questions
Where do I put robots.txt?
In the root of your domain, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
Does Disallow remove a page from Google?
No — it only stops crawling. A blocked page can still be indexed if other sites link to it. To keep a page out of search, allow crawling and add a noindex robots meta tag.
How do I block AI crawlers?
Use the “Block AI training crawlers” template. It disallows GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and Applebot-Extended while normal search engines can still crawl.
How does the tester decide?
Like Google: the most specific user-agent group applies, the longest matching rule wins, and Allow wins a tie. * matches anything and $ marks the end of the URL.
Is anything sent to your server?
Only when you load a live robots.txt: our server fetches that file for you. Rules you write and test stay in your browser.