The Complete Guide to Robots Txt

Key takeaways

• A robots txt file controls crawling, not indexing. A blocked page can still appear in results if other sites link to it, so use meta noindex when you need a page kept out of the index.
• The file lives at one exact location: yourdomain.com/robots.txt. Placing it anywhere else means crawlers never find it and treat the whole site as open.
• Disallow rules match by path prefix and are case-sensitive. A rule for /Admin does nothing to /admin, which is how sensitive folders leak into crawls.
• Always test changes in Google Search Console before deploying. One misplaced slash in a robots txt file can block your homepage across every search engine at once.

What a robots txt file does and where it lives

A robots txt file is a set of instructions for automated crawlers, served as plain text from the root of your domain. When a well-behaved bot like Googlebot visits, it requests yourdomain.com/robots.txt before crawling anything else and reads the rules there to decide which URLs it is allowed to fetch. The file has to sit at the root. A robots txt file placed in a subfolder is invisible to crawlers, which then assume every path is open to them.

The format is deliberately plain. You declare a user-agent, then list the paths that agent may not crawl. There is no login, no header, and no compilation step, which is part of why it is so easy to break. A file that returns a 500 error or times out can cause some crawlers to pause fetching the whole site until it responds again, so hosting reliability matters as much as the rules inside.

The syntax: user-agent, disallow, and allow

Every block in a robots txt file starts with a User-agent line naming the crawler the rules apply to, followed by one or more Disallow or Allow directives. A star as the user-agent means the block applies to all crawlers. An empty Disallow value means nothing is blocked, while Disallow: / blocks the entire site, a two-character difference that has taken more than one launch offline.

Here is the core vocabulary:

• User-agent: names the bot, or star for all bots
• Disallow: a path prefix the bot may not crawl
• Allow: carves an exception out of a broader Disallow
• Sitemap: points crawlers to your XML sitemap URL

Rules are matched by prefix and are case-sensitive, so plan paths carefully rather than assuming the crawler will guess your intent.

Crawling versus indexing: the distinction that trips people up

The most common misunderstanding is that a robots txt file keeps a page out of Google. It does not. It stops crawlers from fetching the page content, but if another site links to that URL, Google can still list it in results, often with an empty “no information available” description because the crawler was never allowed to read the page. Blocking a URL you want hidden can make it more visible, not less.

When your goal is to keep a page out of the index entirely, the right tool is a meta robots noindex tag or an X-Robots-Tag HTTP header, and the page must stay crawlable so the bot can see that instruction. If you block the page in robots txt and add noindex, the crawler never reads the noindex, and the block quietly defeats itself. Decide first whether you are managing crawl budget or search visibility, because the two problems need different fixes.

Managing crawl budget on large sites

On a site with a few dozen pages, crawl budget is a non-issue and a robots txt file barely earns its keep. On an e-commerce catalog with faceted navigation, the math changes fast. Filter combinations can generate hundreds of thousands of near-duplicate URLs, and every one a crawler fetches is one it did not spend on a page you care about. A robots txt file that blocks parameter-heavy filter paths keeps the crawler focused on canonical product and category pages.

The tradeoff is precision. Block too aggressively and you can cut off URLs that feed internal linking or that you later want indexed. The safer pattern is to identify the specific query parameters that create duplication, block those path patterns, and confirm in your server logs that crawler hits shift toward the pages that drive revenue. Crawl budget management is measurement work, not a one-line fix you set and forget.

Common robots txt mistakes that cost rankings

The mistakes that do real damage are rarely exotic. The most destructive is shipping Disallow: / from a staging environment into production, which tells every crawler to leave the entire site alone. It often survives a deploy unnoticed because pages already in the index stay ranked for a while before dropping. By the time traffic falls, the cause is weeks old.

Other frequent errors include blocking your CSS and JavaScript folders, which stops Google from rendering the page the way users see it and can hurt mobile evaluation, and assuming a Disallow hides confidential files when the file itself is public and readable by anyone. A robots txt file is not a security control. If a path must stay private, protect it with authentication, not a Disallow line that effectively publishes a list of the directories you would rather people not visit.

Wildcards, path matching, and the sitemap directive

Google and most major crawlers support two pattern characters in a robots txt file: the star, which matches any sequence of characters, and the dollar sign, which anchors a match to the end of a URL. Disallow: /*.pdf$ blocks every URL ending in .pdf, while Disallow: /*? blocks any URL containing a query string. These are powerful and easy to over-apply, so test each pattern against real URLs before trusting it.

The Sitemap directive is the one line that adds rather than restricts. Listing your XML sitemap URL in the robots txt file gives crawlers a direct route to every page you want discovered, independent of your internal linking. It is a good habit even when you submit sitemaps through Search Console, because not every crawler that reads your file uses Google’s tooling. Keep the sitemap URL absolute and current whenever the file is updated.

Testing and validating before you deploy

Never edit a live robots txt file by hand and hope. Google Search Console includes a robots txt report that shows the fetched file, flags syntax problems, and lets you check whether a specific URL is allowed or blocked for Googlebot. Run every important template through it: your homepage, a top category, a product page, a blog post, and any path you intended to block. If the tool says a page you need indexed is blocked, you caught the problem before the crawler did.

Build the check into your release process. Treat the robots txt file as code, keep it in version control, and diff it on every deploy so a change from staging cannot slip through unseen. For larger sites, a scheduled monitor that fetches the file daily and alerts on any change to the Disallow rules turns a catastrophic mistake into a five-minute fix instead of a month-long mystery.

Robots txt for staging, subdomains, and multiple bots

Each subdomain and each protocol needs its own robots txt file. The rules at blog.yourdomain.com/robots.txt are entirely separate from those at the root domain, and http and https are treated as distinct origins. Staging environments should serve a file that blocks everything, but the danger is that this exact file gets promoted to production during a deploy, so environment-specific configuration matters more than a single shared file.

You can also write different rules for different crawlers by declaring multiple user-agent blocks. You might allow Googlebot full access while blocking an aggressive scraper or an AI training bot by name. Order and specificity follow each crawler’s own matching logic, and Google uses the most specific matching group, so keep the blocks clearly separated and comment them so the next person understands why each rule exists.

Fitting robots txt into a wider technical SEO plan

A robots txt file is one lever among several: canonical tags, meta robots directives, XML sitemaps, and internal linking all shape how search engines crawl and index a site, and they have to agree with each other. A page blocked in robots txt but marked canonical elsewhere sends mixed signals, and crawlers resolve that ambiguity in ways you may not like. Audit these controls together rather than tuning one in isolation.

If you want a technical audit that checks your robots txt file alongside your sitemap, canonicals, and crawl logs, our team can review your setup and prioritize the fixes that move indexing. Browse our related SEO guides to keep building the picture before you make changes to a live file.

More seo & content guides