Allow all crawlers
Open site for indexing.
Preset: Allow allUser-agent: *
Disallow:
Create a robots.txt file with User-agent rules, Allow/Disallow paths, Sitemap URLs, and optional Crawl-delay - instantly in your browser.
Choose a preset or define custom Allow/Disallow paths, then add Sitemap URLs.
Use * for all crawlers, or a specific bot name
One path per line (e.g. /admin/)
One path per line; useful for exceptions under a blocked folder
One absolute URL per line
Seconds; ignored by Googlebot
Mainly used by Yandex
One tap, no account. Your vote is saved in this browser and counted in our site analytics.
Allow all, block all, or build custom Allow/Disallow rules in seconds.
Add one or more Sitemap URLs and an optional Host directive.
See the robots.txt update as you edit paths, agents, and options.
Paste the file at your site root or save robots.txt for deploy.
Start with Allow all, Block all, or Custom rules for full control.
Use * for all bots, or target a specific crawler. Add Disallow/Allow paths one per line.
Link your XML sitemap(s). Optionally set Crawl-delay or Host when needed.
Place the file at https://your-domain.com/robots.txt and verify in Search Console.
Generate a valid robots.txt with a sitemap reference before a new site goes live and gets crawled for the first time.
Create a disallow-all rule for a staging or preview domain so it does not get indexed alongside the production site.
Update crawl rules to disallow old private paths and point crawlers at a refreshed sitemap after a site restructure.
Set custom Allow and Disallow paths for one named user-agent while leaving default access for other bots unchanged.
Open site for indexing.
Preset: Allow allUser-agent: *
Disallow:
Hide admin and draft paths.
Disallow: /admin/
Disallow: /private/User-agent: *
Disallow: /admin/
Disallow: /private/Point crawlers at your XML sitemap.
Sitemap: https://example.com/sitemap.xmlSitemap: https://example.com/sitemap.xmlAll rules are generated locally in your browser. Nothing is uploaded to a server.
A robots.txt generator builds the plain-text crawl policy file that tells bots which paths they may fetch, and can point them to your sitemap.
It helps site owners create consistent, valid robots.txt rules quickly and avoid syntax mistakes that block indexing.
It does not crawl your site, verify live robots.txt, or guarantee crawler compliance. Sensitive content still needs proper access control.
Meta Tags That Actually Help SEO
Title and description best practices, Open Graph basics, what robots.txt really does, and common SEO meta myths worth dropping.
Why Browser-Based Tools Are Often More Private
How client-side processing keeps files on your device, when servers are still necessary, and how ToolMint approaches privacy for everyday utilities.
See every utility on the all tools page.
Found a problem or missing feature? Tell us - no account required.
Last updated: