Robots.txt Generator & Tester
Create compliant robots.txt rules visually and verify crawl allowances with the live path tester.
Define Robots Rules
Add Custom Directive
Current Custom Rules (2)
*Disallow/private/
*Disallow/api/
Generated robots.txt
# Generated by DevUtils Lab (https://www.devutilslab.dev)
User-agent: *
Disallow: /private/
Disallow: /api/
Sitemap: https://www.devutilslab.dev/sitemap.xmlLive Path Tester
Understanding Robots.txt
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page off Google. To keep a web page off Google, use noindex directives instead.
How to use
- 1Select default crawler access (Allow All / Disallow All).
- 2Add custom directives for specific user-agents (e.g. Googlebot, Bingbot) and paths.
- 3Use the Live Path Tester to input an agent and route path to verify the crawl resolution.
- 4Copy the generated robots.txt output and upload it to your web server's root folder.