Robots.txt Generator

How to Use

Choose a default access mode, add any paths you want to block or specifically allow, and optionally link your sitemap. The generated file updates live below. Download it and upload it to your website's root directory so it's reachable at yourdomain.com/robots.txt. If you only need a quick "allow everything" or "block everything" file, the first two default access options handle that without touching the path fields at all, the custom option is only necessary when you want fine-grained control over specific folders.

What robots.txt Actually Does

robots.txt is a plain text file that follows the Robots Exclusion Protocol, a voluntary standard that well-behaved crawlers check before fetching any page on your site. It sits at your domain's root and tells crawlers which parts of the site they're welcome to request and which parts they should skip. The key word is voluntary: robots.txt is a request, not a lock. Major search engines like Google, Bing, and most legitimate crawlers respect it, but nothing technically prevents a crawler from ignoring it, so it should never be treated as a security or access-control mechanism for genuinely sensitive content. Anything you truly need to keep private belongs behind authentication, not behind a Disallow rule.

Worked Example: A Typical E-Commerce robots.txt

Picture an online store with an admin panel at /admin/, a shopping cart at /cart/, internal search results at /search?, and a public product catalog everywhere else. Selecting "Custom" and listing /admin/, /cart/, and /search? as Disallow paths, with a Sitemap URL of https://example.com/sitemap.xml, produces:

User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search? Sitemap: https://example.com/sitemap.xml

This keeps crawlers away from pages that offer no search value (an empty cart, a login screen, infinite search-query permutations) while leaving every product and category page fully crawlable, and it points crawlers straight at the sitemap so they can discover new products efficiently instead of relying purely on internal links.

Common Mistakes That Accidentally Deindex a Whole Site

The single most damaging robots.txt mistake is accidentally shipping Disallow: / to a live production site, usually because a rule meant only for a staging environment never got removed before launch. This one line tells every well-behaved crawler to stop fetching anything on the domain, and traffic can quietly drop to near zero over the following weeks as previously indexed pages age out of the index. Always double-check the generated output before uploading, especially after switching the "Default Access" dropdown, since selecting "Block all crawlers" is only one click away from "Allow all crawlers." A second common mistake is disallowing a path that contains resources the page itself needs to render properly, like a shared /assets/ or /css/ folder, which can cause Google to see a broken, unstyled version of your pages during rendering and mis-judge their quality.

Allow Rules and Disallow Rules Together

The Allow directive exists to carve out an exception inside a broader Disallow rule, rather than to grant access on its own, most crawlers already allow anything you haven't explicitly disallowed. A practical case: you disallow /private/ entirely, but want one specific public page inside it, say /private/press-kit.html, to remain crawlable. Listing /private/ under Disallow and /private/press-kit.html under Allow does exactly that. When rules conflict, most crawlers, including Googlebot, apply the more specific (longest) matching path rather than just reading rules top to bottom, so a longer Allow path will win over a shorter Disallow path that also matches.

robots.txt vs Meta Robots Tag vs X-Robots-Tag

These three tools sound similar but operate at different layers and are frequently confused. robots.txt (this tool) is a single site-wide file that controls crawling, whether a crawler is permitted to request a URL at all. The meta robots tag, generated by this site's Meta Tag Generator, lives inside a specific page's HTML and controls indexing and link-following for that one page, but only works if the crawler was allowed to fetch the page in the first place. The X-Robots-Tag is a lesser-known third option, an HTTP response header, useful for controlling indexing on non-HTML files like PDFs and images where there's no HTML head to put a meta tag in. A frequent mix-up is disallowing a page in robots.txt while also trying to noindex it with a meta tag, since the crawler can never reach the page to read the noindex instruction, the disallow silently wins and the meta tag never gets a chance to apply.

Frequently Asked Questions

Where do I upload the generated robots.txt file?

It needs to sit at the root of your domain, so it's reachable at https://yourdomain.com/robots.txt exactly, not inside a subfolder. Most hosting platforms let you upload it directly to the site's root directory.

Does robots.txt actually stop pages from appearing in Google?

Not reliably by itself. A Disallow rule only asks crawlers not to fetch a page's content, but the URL can still appear in search results (usually without a description) if other pages link to it. To truly keep a page out of search results, use a noindex meta tag on the page itself, which requires crawlers to be able to access it first.

Can I block one specific crawler like GPTBot without blocking Google?

Yes, robots.txt supports multiple User-agent blocks in the same file. A block starting with User-agent: GPTBot followed by Disallow: / only applies to that crawler, while a separate User-agent: * block still governs everyone else. This tool generates a single User-agent: * block for simplicity, add extra named blocks manually below the generated output if you need per-crawler rules.

What does an empty Disallow line mean versus Disallow: /?

An empty value, written as just Disallow: with nothing after the colon, means nothing is blocked and crawlers may access the entire site. Disallow: / with a single forward slash blocks the entire site starting from the root. The two look similar but produce opposite outcomes, so it's worth double-checking which one your generated file contains before uploading it.

Is robots.txt case-sensitive?

Yes. Paths in Disallow and Allow rules are matched exactly as written, including case, so Disallow: /Admin/ will not block a URL at /admin/. Directive names like User-agent, Disallow, and Sitemap are conventionally capitalized this way but crawlers generally accept them regardless of case, it's the path values themselves where case sensitivity actually causes real-world mistakes.

Will blocking a path in robots.txt remove it from Google if it's already indexed?

No, and this is a common misunderstanding. Blocking a URL in robots.txt only prevents future crawling, it does not remove an already-indexed page. To remove an indexed page, first let Google crawl it with a noindex meta tag in place so it can read the instruction, or use the URL removal tool in Google Search Console for a faster temporary removal.