Home › Toolkit › Developer › Robots.txt Generator

🤖 Robots.txt Generator

One small file tells every crawler where it may and may not go. Generate yours properly: pick the agent, list the paths to keep private, and add the sitemap line engines love.

About this tool

robots.txt is a politeness protocol with real consequences: well-behaved crawlers read it before touching your site, and follow its Disallow rules. This generator produces a clean, correctly structured file — user-agent groups, their rules in order, blank lines where parsers expect them, and the Sitemap line that tells engines exactly where your page list lives. Small file, but a malformed one can accidentally hide a whole site from search, which is why generating beats hand-typing.

Understand its limits before you lean on it. robots.txt is not security: it asks, it does not enforce, and it publicly lists the paths you name — advertising /admin/ to every curious reader while keeping only the polite bots out. Real protection is authentication. What robots.txt genuinely controls is crawl budget and politeness: keeping crawlers out of infinite calendar pages, internal search results and staging areas so they spend their visits on pages that matter.

How to use it

  1. Set the user-agent (* for everyone, or a specific bot).
  2. List the paths to disallow, one per line.
  3. Add your sitemap URL and press Generate; save the file as robots.txt in your site's root.

Good to know

  • An empty Disallow (Disallow: with nothing after it) means 'allow everything' — the generator writes it when your list is empty.
  • Test the live file after uploading: visit yourdomain.com/robots.txt and then check it in Google Search Console's tester.
  • Never disallow your CSS/JS folders wholesale — engines need them to render pages, and blocking them can hurt rankings.

Frequently asked questions

Can robots.txt keep a page out of Google entirely?

It stops crawling, but a linked page can still appear in results without a description. For true removal, use a noindex tag on the page itself.

Where does the file go?

In the site's root, so it lives at yourdomain.com/robots.txt — nowhere else is read.

Does every bot obey it?

Reputable crawlers do. Rogue scrapers ignore it, which is why it is a crawl-management tool, not a security tool.

More Developer tools

Making videos too? Try our YouTube SEO Checker, or browse the full Toolkit and the free YouTube tools.