How to Write a Robots.txt File That Doesn’t Block Your Whole Site

October 17, 2026 • NOFYI Editorial Team

A robots.txt file is one of the smallest files on a website, and also one of the easiest to get wrong. With a single misplaced slash, you can accidentally tell search engines to stay away from your entire site. With a careful file, you can guide crawlers toward the pages that matter and away from the clutter that does not need to be crawled. This guide explains what robots.txt does, how to write it safely, and how to test it before it causes problems.

You do not need to be an expert to work with robots.txt, but you do need to be precise. The file uses a very simple syntax, and search engines read it literally. That is why a small typo matters more here than in most other places on your site. The good news is that once you understand the two main rules — User-agent and Disallow — and the habit of testing every change, robots.txt becomes a straightforward part of basic site maintenance.

What Is Robots.txt?

Robots.txt is a plain text file that lives at the root of your website, for example at example.com/robots.txt. It gives instructions to web crawlers, which are automated programs used by search engines to discover and fetch pages. The instructions say which parts of the site a crawler may or may not request. The file is a request, not a lock. Well-behaved crawlers, such as Googlebot, follow it. Other bots may ignore it, and it does nothing to stop a human visitor from opening a page directly.

The file is public. Anyone can open your robots.txt file in a browser and read it. That has an important consequence: you should never use robots.txt to hide sensitive information. If you list a private folder in a Disallow rule, you are effectively advertising that the folder exists. For anything that must stay private, use proper access controls, passwords, or authentication, not robots.txt.

Robots.txt is part of a standard called the Robots Exclusion Protocol. It has been used for many years, and the basic idea has stayed stable. You create groups of rules. Each group starts by naming which crawler it applies to, using a User-agent line. Then you list the paths that crawler should not fetch, using Disallow lines. You can also list paths that are explicitly allowed with Allow lines, and you can point to your XML sitemap with a Sitemap line. That is most of what you need to know.

The Basic Syntax, Explained

A simple robots.txt file might look like this in words: for all crawlers, do not fetch the admin folder, but do allow everything else. In code, that is written as User-agent: * on one line, Disallow: /admin/ on the next line, and Allow: / on a third line. The asterisk means all crawlers. The slash in /admin/ means the folder named admin at the root of the site. The trailing slash matters. Without it, the rule could match more than you intend.

Each rule is about URL paths, not about whole websites. Disallow: /admin/ blocks URLs that start with /admin/, such as /admin/settings/ or /admin/users/. It does not block a page such as /about-admin/ unless the path actually starts with the blocked prefix. This prefix matching is the reason a single character can have a large effect. Disallow: / blocks everything because every path starts with a slash. Disallow: /admin blocks any path starting with /admin, including /admin and /administrator if that path exists. Precision matters.

You can also use a few special patterns. An asterisk can stand in for any sequence of characters, and a dollar sign at the end of a path means the path must end there. For example, Disallow: /*?sort= could block URLs that contain a sort parameter. These patterns are useful, but they also increase the risk of accidental over-blocking. For most small sites, simple prefix rules are enough. Start simple, test carefully, and only add patterns when you have a clear reason.

How to Write a Safe Robots.txt File

Begin by deciding what you actually need to block. For many sites, the answer is very little. You might want to block an admin area, a search results page that generates endless duplicate URLs, a staging folder, or a directory of temporary files. You usually do not need to block your main content, your blog posts, your product pages, or your category pages. If in doubt, allow it. It is safer to let a crawler fetch a harmless page than to accidentally block an important one.

Next, write the file in a plain text editor, not in a word processor that might add hidden formatting. Use one rule per line, keep the file short, and add comments with a hash symbol if you want to remind yourself why a rule exists. Comments are lines that start with #. They do not affect crawlers, but they help future you or a colleague understand the intent. For example, you might write a comment explaining that a particular folder is blocked because it contains internal tools.

Place the file at the root of your domain so that it is available at yourdomain.com/robots.txt. If your site uses a content management system, there may be a setting or plugin that manages robots.txt for you. In WordPress, for example, some SEO plugins let you edit robots.txt from the dashboard. If you manage the file manually, upload it to the root and then open the URL in a browser to confirm it loads. If it returns a 404 error, crawlers will assume there is no file and will crawl normally, which is not ideal if you intended to block something.

Worked Example: A Small Site That Blocks Only Its Admin Area

Imagine you run a small online magazine with three sections: a homepage, a set of articles, and an admin area where editors log in and manage content. You want search engines to crawl the homepage and the articles, but you do not want them to waste time crawling the admin area. You also want to point them to your XML sitemap. A safe robots.txt file for that site could be written as follows in plain form.

The file would start with User-agent: * to apply the rules to all crawlers. The next line would be Disallow: /admin/ to block the admin folder. The next line would be Allow: / to make clear that everything else is allowed. The final line would be Sitemap: https://example.com/sitemap.xml, using your real domain and sitemap address. That is the whole file. It is short, it is clear, and it does exactly what the site owner intended.

Now consider what happens if a single character changes. If the file says Disallow: / instead of Disallow: /admin/, the rule now blocks every path on the site. Search engines that follow the file would stop crawling the homepage, the articles, and everything else. That is the classic one-slash mistake. The file would still look almost the same at a glance, which is why testing matters. A quick test after every edit can catch this error before it affects search visibility.

Another common mistake is to write Disallow: /admin without the trailing slash when you only meant to block the folder. As noted earlier, that rule would also block any URL that starts with /admin, which might include a public page such as /admin-guide/ if one existed. The fix is simple: use the full folder path with the trailing slash, /admin/, when you mean a folder. If you mean a single file, use its full path, such as /private-file.pdf. Being specific prevents surprises.

Before publishing any robots.txt change, read the file out loud as a set of instructions. For each Disallow line, ask: which URLs does this actually block, and do I want all of them blocked? For each Allow line, ask: does this override a block I intended to keep? That slow reading step takes less than a minute and catches many errors that a quick glance misses.

How to Test Robots.txt in Search Console

Google Search Console offers a robots.txt tester that lets you check whether a specific URL is allowed or blocked. To use it, sign in to Search Console, select your property, and open the robots.txt report or tester. The tool shows the current robots.txt file that Google has fetched. You can enter a URL to test and see whether Googlebot would be allowed to fetch it. You can also edit the file in the tester to preview changes before you publish them.

Test the most important URLs on your site: the homepage, a key article or product page, a category page, and any URL you intended to block. For the blocked URLs, the tester should say they are blocked. For the allowed URLs, it should say they are allowed. If the results do not match your intent, do not publish the change. Fix the file, test again, and only then upload the corrected version. After publishing, ask Search Console to recrawl or simply wait for Google to fetch the updated file.

It is also useful to check the tester after a site migration, a redesign, or a change of hosting. Those events can reset or overwrite robots.txt without you realising it. A quick check during those moments can prevent a situation where your new site is accidentally blocked for days or weeks. Make robots.txt testing part of your launch checklist, alongside checking the sitemap, the homepage, and the main navigation.

Common Robots.txt Mistakes to Avoid

The most dangerous mistake is blocking everything by accident. That usually happens with Disallow: / or with a wildcard pattern that is broader than intended. Always search your file for a bare Disallow: / line before publishing. If you see it, make sure you truly mean to block the whole site. In most cases, you do not.

Another mistake is blocking CSS, JavaScript, or image files that Google needs to render your pages. In the past, some site owners blocked those resources to save crawl budget. Today, blocking them can prevent Google from understanding how your page looks and works. Unless you have a specific technical reason, allow crawlers to fetch the assets that your pages need.

A third mistake is using robots.txt to try to keep a page out of search results. Blocking a page in robots.txt does not guarantee that it will not appear in search results. Google may still show the URL if it finds links to it elsewhere, even if it cannot fetch the content. If you want a page to stay out of the index, use a noindex directive or proper access control instead of relying on robots.txt alone. Robots.txt controls crawling, not indexing, and the two are not the same.

Finally, avoid duplicating rules for many different crawlers unless you need to. A single group with User-agent: * covers all crawlers. Only add separate groups when you have a clear reason to treat one crawler differently, such as allowing a specific bot to fetch a folder that others should not. Extra groups add complexity and increase the chance of a conflicting rule.

Limitations: Robots.txt Is Not Security

Robots.txt is not a security tool. It does not require a login, it does not encrypt anything, and it does not stop a determined visitor or a badly behaved bot from requesting a page. It is a polite request that well-behaved crawlers follow. If a page contains sensitive data, private user information, or internal tools, you need authentication, permissions, and other security controls. Do not rely on robots.txt to protect it.

Robots.txt also does not control how search engines rank your pages. Allowing a page to be crawled does not make it rank well, and blocking a page does not make other pages rank better. Ranking depends on content quality, relevance, links, and user experience. Robots.txt is about crawl access, not about quality signals.

There are also limits to how much you should try to fine-tune crawling. For most small sites, search engines can crawl the site without special help. Overly complex robots.txt files can do more harm than good. Keep the file simple, keep it accurate, and focus most of your effort on content and site structure. Robots.txt should support that work, not replace it.

FAQ

Where should robots.txt live?

It should live at the root of your domain, for example at yourdomain.com/robots.txt. Crawlers look there by default. If it is in a subfolder, crawlers may not find it.

Does robots.txt block people from visiting a page?

No. It only gives instructions to crawlers. A human visitor can still open a blocked URL directly in a browser. If you need to restrict people, use passwords or permissions.

What does Disallow: / mean?

It blocks all crawling, because every URL path starts with a slash. Use it only if you truly want to block the entire site, such as for a staging site that should not be crawled. Double-check it before publishing.

How do I know if my robots.txt is working?

Use the robots.txt tester in Google Search Console to check specific URLs, and open the file in a browser to confirm it loads. After publishing, monitor Search Console for crawl errors or unexpected drops in indexed pages.

Conclusion

A good robots.txt file is short, specific, and tested. Start by blocking only what you truly need to block, write the rules with exact paths, and use Allow: / to make the default clear. Test every change in Search Console before and after publishing, and remember that robots.txt is a guide for crawlers, not a security barrier. With that careful approach, you can keep search engines crawling the pages that matter without risking an accidental site-wide block.