Robots.txt Generator
Create a robots.txt file to control search engine crawling
About this ToolHow it works, benefits & use casesTap to collapse
This Robots.txt Generator builds a valid robots.txt file from a visual rule editor instead of hand-typing directives. Each rule is a user-agent block where you set the agent name (such as * or Googlebot) and list Allow and Disallow paths as comma-separated values, and you can add as many blocks as you need with the Add rule button. Three presets fill the editor instantly: Allow All opens the whole site to every crawler, Disallow All blocks everything, and Google Only permits Googlebot while disallowing all other agents. A separate field appends an optional Sitemap line pointing crawlers to your sitemap.xml, and that sitemap URL is preserved when you switch presets. Click Generate to render the file, with a line count shown, then copy or download it as robots.txt. Your configuration is saved automatically in the browser.
How to Use
- 1Start from a preset — Allow All, Disallow All, or Google Only — or build rules from scratch.
- 2For each user-agent block, set the agent name and enter Allow and Disallow paths as comma-separated lists.
- 3Use Add rule to create additional user-agent blocks, or the trash icon to remove one (the last block stays).
- 4Optionally add a Sitemap URL; it is appended as a Sitemap line and kept when you change presets.
- 5Click "Generate robots.txt", then copy the result or download it as robots.txt to your site root.
Key Benefits
- Visual multi-block editor instead of memorizing robots.txt syntax
- Three ready presets: Allow All, Disallow All, and Google Only
- Comma-separated Allow and Disallow path entry per user-agent
- Optional Sitemap line that survives preset switches
- Line count and a built-in directive quick-reference
- Configuration auto-saved in your browser between visits
- Copy or download the file directly as robots.txt — all in-browser
Common Use Cases
- Blocking admin or private paths from search engine indexing
- Quickly opening a site to all crawlers after a staging lockdown
- Restricting crawling to Googlebot only during a controlled rollout
- Adding a sitemap reference so crawlers discover your URLs faster
- Generating distinct rules for different bots in a single file
Presets
User-agent rules
Run to generate.
Configure rules above and click Generate.
Quick reference
User-agent: *— applies to all crawlersDisallow: /admin— block a pathAllow: /public— explicitly allow within a blocked sectionSitemap:— point crawlers to your sitemap
Was this tool helpful?
Share Your Experience
Help others discover this tool!
Related guide
Related tools
- Changelog GeneratorGenerate changelog from Git commit history
- Component Name GeneratorTurn a description into ranked, kind-aware component names with a file scaffold and a name validator (casing, collisions, clarity)
- Heroku Procfile GeneratorBuild a Heroku Procfile and matching app.json — process types, env vars, addons, dyno formation — plus a Procfile parser and audit
- Markdown TOC GeneratorGenerate table of contents for Markdown
- Package.json Scripts GeneratorGenerate common npm scripts for different project types and workflows
- Tailwind Config GeneratorVisually build a Tailwind config - theme extend, colors, spacing, fonts, screens, plugins - and export it as a v3 JS config or a v4 @theme CSS block
Allow All creates a single User-agent: * block that allows /, opening the whole site to every crawler. Disallow All blocks the entire site with Disallow: / for all agents. Google Only allows Googlebot to crawl everything while disallowing all other user agents.
The Allow and Disallow fields accept comma-separated paths — for example "/admin, /private" produces two separate Disallow lines. Whitespace around each path is trimmed and empty entries are ignored, so only real paths end up in the file.