What Is Robots.txt and Should You Care

October 17, 2026 • NOFYI Editorial Team

If you run a website, you may have heard that you need a robots.txt file. If you do not run a website, the name may sound like a robot instruction manual. In fact, robots.txt is a small text file that sits at the root of a website and gives polite instructions to search engines and other automated visitors, called crawlers or bots. It does not control people. It does not protect secrets. It simply asks well-behaved bots which parts of your site they should and should not visit. This guide explains what each line means, how to write a safe file, and when you should care.

Think of a website as a building with public rooms, staff rooms, and storage cupboards. A robots.txt file is a sign at the front door that says: visitors are welcome in the public rooms, please do not go into the staff rooms, and here is a map of the building. A polite visitor will follow the sign. A rude visitor may ignore it. That distinction is the key to understanding the file. Robots.txt is a request, not a lock.

How to Understand Robots.txt

Every robots.txt file is made of groups. Each group starts with one or more User-agent lines, followed by rules. A User-agent is the name of a bot. A star means all bots. For example, User-agent: * means the rules that follow apply to every bot that reads the file. User-agent: Googlebot means the rules apply only to Google’s crawler. If you want different rules for different bots, you write separate groups.

After the User-agent line come the rules. The two most important are Disallow and Allow. Disallow tells a bot not to crawl a path. Allow tells a bot that it may crawl a path, even if a broader Disallow would otherwise cover it. Paths start with a slash and are matched from the start. For example, Disallow: /admin/ tells bots not to crawl anything under the admin folder. Disallow: /private means do not crawl any path that starts with /private.

The third common line is Sitemap. It tells bots where to find your XML sitemap, which is a list of pages you want search engines to know about. A typical sitemap line looks like Sitemap: https://example.com/sitemap.xml. It is usually placed at the end of the file, outside any single group, so that all bots can see it.

Order matters in a useful way. Within a group, a bot generally follows the most specific rule that matches. That is why Allow is useful. You can disallow a whole folder but allow one useful subfolder or file inside it. Without Allow, you would have to list every exception by hand.

How to Write a Basic Robots.txt File

Step one is to decide what you actually want to block. Most small sites need to block very little. Common candidates are admin pages, login pages, internal search results, cart and checkout pages, and duplicate or low-value pages created by filters. Do not block your main content, your blog posts, your product pages, or your images if you want them to appear in search.

Step two is to write the file in plain text, using UTF-8 encoding, with one rule per line. Do not use a word processor that adds formatting. Use a plain text editor or your content management system. Save the file with the exact name robots.txt, all lower case.

Step three is to place the file at the root of your domain. That means it should be reachable at https://your-site.com/robots.txt. A file at https://your-site.com/blog/robots.txt will not work for the whole site. Search engines look only at the root.

Step four is to test the file. Most search engines provide a robots.txt tester in their webmaster tools. Use it to check that important pages are allowed and that blocked pages are blocked. Also open the file in a browser and read it yourself. A single typo, such as Disallow: / with no path after it, can block the whole site.

Worked Example: Allow Images, Block Admin

Here is a complete worked example that matches a common need. Suppose you have a site with an admin area you do not want crawled, an images folder you do want crawled, and a sitemap. A safe file would be:

User-agent: *
Allow: /images/
Disallow: /admin/
Sitemap: https://nofyi.com/sitemap.xml

Let us read it line by line. User-agent: * means these rules apply to all bots. Allow: /images/ means bots may crawl anything under the images folder. Disallow: /admin/ means bots should not crawl anything under the admin folder. Sitemap: https://nofyi.com/sitemap.xml tells bots where the sitemap lives. Replace the domain with your own domain.

Now suppose your admin folder also contains a public help page at /admin/help that you do want crawled. You could add a more specific Allow line for that page, and most bots would honour the more specific rule. That is the power of combining Allow and Disallow. You do not need to move the page. You just add a precise exception.

A different example shows a common mistake. If you write Disallow: / with no path after the slash, you are telling all bots not to crawl anything. That is useful only if you want to hide a staging site from search engines. On a live site, it is a disaster. Always double-check a bare Disallow: / line. If you see it on a live site, remove it or narrow it.

What Robots.txt Does Not Do

Robots.txt does not make a page private. It does not require a login. It does not encrypt anything. It does not stop a person from visiting a URL if they know it or guess it. It does not stop a bot that ignores the file. Bad bots, scrapers, and attackers routinely ignore robots.txt. Some even read it to discover which folders a site owner considers sensitive, and then visit those folders anyway.

Robots.txt also does not remove a page from search results by itself. If a page is linked from elsewhere, a search engine may still show its URL even if crawling is disallowed, because the engine knows the URL exists but has not been allowed to read the content. If you need a page to stay out of search results, use a noindex tag or require a login, not only a Disallow rule. If you need a page to stay secret, protect it with authentication and do not link to it.

Finally, robots.txt does not control how your site appears in search. It controls which parts bots are asked to crawl. Ranking, titles, and descriptions are influenced by content, links, and many other factors. A correct robots.txt file helps crawlers spend their time on useful pages, but it will not make a thin or unhelpful page rank well.

Should You Care?

For most small sites, the answer is: care a little, but do not obsess. A simple, correct robots.txt file is worth having. It keeps bots out of admin and internal pages, it points them to your sitemap, and it helps search engines use their crawl time wisely. A missing file is not a crisis, because bots will simply crawl what they can find. A wrong file is a bigger risk than a missing file, because it can block important pages.

You should care more if your site is large, if it has many filtered or duplicate pages, if it has a staging copy, or if you see crawl errors in search tools. In those cases, a thoughtful robots.txt file saves server resources and prevents duplicate pages from crowding out useful ones. You should also care if you run an online shop, because cart, checkout, and account pages should not be crawled, and because faceted navigation can create thousands of near-duplicate URLs.

If you are a reader rather than a site owner, robots.txt is still useful to know about. It explains why a search engine may not have indexed a page, and it reminds you that search results are not a complete view of the web. Some pages are intentionally not crawled, and some pages are private even if they are technically reachable.

Limitations

Robots.txt is limited by design. It relies on voluntary compliance. It has no enforcement. It does not distinguish between a good bot and a bad bot that claims to be good. It does not handle complex rules well, and different bots may interpret edge cases slightly differently. Wildcards and pattern matching are supported by major search engines, but not by every bot, so keep your rules simple and explicit.

The file is also public. Anyone can open https://your-site.com/robots.txt and read it. Do not list secret paths there and assume they will stay secret. If a path must stay hidden, protect it properly and do not advertise it in robots.txt. Use Disallow only for paths that are already protected or that are simply not useful for search.

Another limitation is caching. Bots may cache your robots.txt file for hours or days. If you fix a mistake, search engines may not notice immediately. That is another reason to test before publishing and to avoid blocking your whole site by accident.

Frequently Asked Questions

Where does robots.txt live?

At the root of your domain, for example https://your-site.com/robots.txt. Search engines look there and only there. A file in a subfolder does not apply to the whole site.

Does robots.txt keep a page secret?

No. It is a request, not a lock. Anyone who knows the URL can visit it, and bots that ignore the file can crawl it. Use login protection and noindex for pages that must stay private or out of search.

What is the difference between Allow and Disallow?

Disallow asks bots not to crawl a path. Allow explicitly permits a path, and it can override a broader Disallow when it is more specific. Use Allow to create narrow exceptions, such as allowing one public page inside an otherwise blocked folder.

Do I need a Sitemap line?

It is helpful but not required. A Sitemap line tells bots where your sitemap is without them having to guess. If you have a sitemap, adding the line is a simple way to help crawlers find it.

Robots.txt is a small file with a clear job. It tells polite bots where to go, where not to go, and where to find your sitemap. Use User-agent to choose who the rules apply to, Disallow to block what is not useful, Allow to make precise exceptions, and a Sitemap line to guide crawlers. Remember that it is a request, not a lock, and keep secret pages protected by proper login, not by a text file. Done that way, robots.txt is a quiet helper, not a source of worry.