Free Robots.txt Generator
Build a valid robots.txt file with custom crawler rules.
robots.txt controls crawler access, not search indexing directly — a disallowed page can still appear in search results (without a snippet) if other pages link to it. To fully prevent indexing, use a noindex meta tag on the page itself instead of (or in addition to) robots.txt.
Worked Example
Example: Disallowing /admin/ and /cart/ with a sitemap reference produces a ready-to-upload robots.txt for your site root.
Method
Standard robots.txt syntax: User-agent: * followed by Disallow:/Allow: path rules and an optional Sitemap: line.
Worked Example
Example: Disallowing /admin/ and /cart/ with a sitemap reference produces a ready-to-upload robots.txt for your site root.
Method
Standard robots.txt syntax: User-agent: * followed by Disallow:/Allow: path rules and an optional Sitemap: line.