About this tool
robots.txt is a politeness protocol with real consequences: well-behaved crawlers read it before touching your site, and follow its Disallow rules. This generator produces a clean, correctly structured file — user-agent groups, their rules in order, blank lines where parsers expect them, and the Sitemap line that tells engines exactly where your page list lives. Small file, but a malformed one can accidentally hide a whole site from search, which is why generating beats hand-typing.
Understand its limits before you lean on it. robots.txt is not security: it asks, it does not enforce, and it publicly lists the paths you name — advertising /admin/ to every curious reader while keeping only the polite bots out. Real protection is authentication. What robots.txt genuinely controls is crawl budget and politeness: keeping crawlers out of infinite calendar pages, internal search results and staging areas so they spend their visits on pages that matter.
How to use it
- Set the user-agent (* for everyone, or a specific bot).
- List the paths to disallow, one per line.
- Add your sitemap URL and press Generate; save the file as robots.txt in your site's root.
Good to know
- An empty Disallow (Disallow: with nothing after it) means 'allow everything' — the generator writes it when your list is empty.
- Test the live file after uploading: visit yourdomain.com/robots.txt and then check it in Google Search Console's tester.
- Never disallow your CSS/JS folders wholesale — engines need them to render pages, and blocking them can hurt rankings.
Frequently asked questions
Can robots.txt keep a page out of Google entirely?
It stops crawling, but a linked page can still appear in results without a description. For true removal, use a noindex tag on the page itself.
Where does the file go?
In the site's root, so it lives at yourdomain.com/robots.txt — nowhere else is read.
Does every bot obey it?
Reputable crawlers do. Rogue scrapers ignore it, which is why it is a crawl-management tool, not a security tool.
More Developer tools
JSON Formatter & Validator
Pretty-print messy JSON, minify it back, and get a precise error location when it will not parse.
JSON to CSV
Turn a JSON array of objects into a clean spreadsheet-ready CSV, columns and escaping handled.
CSV to JSON
Convert spreadsheet CSV into a JSON array — quoted fields, embedded commas and all.
XML Formatter
Indent and validate XML — sitemaps, feeds, config files — with errors pointed at the guilty tag.
JS / CSS / HTML Beautifier
Untangle minified or messy code into readable, consistently indented JavaScript, CSS or HTML.
JS / CSS Minifier
Strip comments and needless whitespace from JavaScript and CSS — smaller files, same behaviour.
Making videos too? Try our YouTube SEO Checker, or browse the full Toolkit and the free YouTube tools.