Robots.txt Generator

Last updated: October 16, 2026  •  Published by: NOFYI Editorial Team  •  Category: SEO Tools

Workspace

What is the Robots.txt Generator?

The Robots.txt Generator is meant to build a robots.txt file for your website. A robots.txt file tells search-engine crawlers which parts of your site they may and may not crawl. You pick a user-agent (which crawler the rules apply to), list the paths to allow or disallow, optionally add your sitemap URL, and the tool composes the plain-text file. You then save that file as robots.txt in your site’s root folder.

How to Use the Robots.txt Generator

  1. Choose a User-agent: All Robots (*) for every crawler, or a specific crawler such as Googlebot or Bingbot.
  2. Check the Allow and Disallow fields and fix the paths to the folders you want to allow or block (see Limitations — the defaults contain double slashes).
  3. Optionally enter your Sitemap URL, for example https://example.com/sitemap.xml.
  4. Click Process and check that a result panel appears with your robots.txt text.
  5. Copy the result and save it as a plain-text file named robots.txt in your website’s root folder.

Worked Example

Intended input: User-agent All Robots (*), Allow /images/, Disallow /admin/ and /private/, Sitemap URL https://example.com/sitemap.xml. A correct robots.txt for those settings looks like this:

User-agent: *
Allow: /images/
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xml

Observed behaviour: in our test, clicking Process with the tool’s default values produced no visible output panel at all — no robots.txt text appeared. Until that is fixed, do not rely on this tool’s output; write the file by hand using the format above and check it before uploading.

Limitations

  • Bug observed: clicking Process with the default values produced no visible output. The tool may currently be broken, so always confirm that a result actually appears before trusting it.
  • The default Allow value is “//images/” and the default Disallow value is “/admin//private/” — both contain double slashes and two paths squeezed into one field. Fix them to clean single paths (“/images/”, “/admin/”, “/private/”) before use.
  • robots.txt is advisory, not access control. Well-behaved crawlers obey it, but it does not block malicious bots or keep private pages secret — use passwords or server rules for that.
  • The tool does not validate your paths, check that your sitemap URL exists, or test the file against a live site.
  • One run produces one user-agent block. If you need different rules for different crawlers, you must combine the blocks yourself.

Frequently Asked Questions

Where do I put the robots.txt file?

In your website’s root folder, so it is reachable at https://example.com/robots.txt. A robots.txt in a subfolder is ignored by crawlers.

Does robots.txt hide pages from Google?

Not reliably. Disallow stops crawling of the path, but a disallowed URL can still be indexed if other pages link to it. To keep a page out of search results, use noindex or password-protect it.

What does “User-agent: *” mean?

It means the rules apply to all crawlers. Choosing Googlebot or Bingbot instead limits the rules to that one crawler.

Does the tool upload or save my file?

No. Everything runs in your browser and nothing is saved between sessions. You must copy the text and upload the file to your own site yourself.

Related Tools

From the Blog