Robots.txt Generator

Generate a robots.txt file with User-agent, Allow, Disallow, and Sitemap directives for basic crawl control.

For this tool, processing runs locally in your browser and data is not sent to the server.

Robots.txt Generator is a free online tool for creating a robots.txt file for your website. You can configure User-agent, Allow, Disallow, Sitemap, and optional Host directives. The tool includes presets for common scenarios and shows warnings for risky rules such as Disallow: /. The tool runs in your browser and does not send data to the server.

What it does

Output

Processing

Continue working

Choose a related tool for the next step.

Data passed between tools is kept only in the current browser session.

How to use

Choose a scenario: allow all crawling, block service sections, block the entire site, or configure rules manually.
Set User-agent, Allow, Disallow, and Sitemap URL.
Review any warnings, especially for Disallow: / or when blocking important site sections.
Copy or download robots.txt and place it in the root of your website.

More about this tool

Robots.txt Generator helps you create a starter robots.txt file for your website. The service lets you configure User-agent, Allow, Disallow, Sitemap, and optionally Host. The tool is suitable for new websites, staging environments, SEO setup, blocking service sections, and drafting a file before publication.

Robots.txt gives instructions to search engine crawlers but does not protect private pages and is not a replacement for authorization, server-level access controls, meta robots, canonical tags, or a full SEO audit. The generator includes presets for common configurations and warns about risky rules.

All processing is done locally in the browser. Your data is not sent to the server.

Runs in the browser
Data is not sent to the server
Generates a robots.txt file
Supports User-agent, Allow, and Disallow
Adds Sitemap
Includes presets for common configurations
Shows warnings for risky rules
Can download the finished robots.txt
No registration required

Frequently Asked Questions

What is robots.txt?

robots.txt is a text file in the root of a website that tells search engine crawlers which sections may or may not be crawled.

Where should robots.txt be placed?

The file is typically placed in the website root, for example at https://example.com/robots.txt.

What does User-agent: * mean?

User-agent: * means the rules apply to all crawlers that honor robots.txt.

What does Disallow do?

Disallow specifies a path that crawlers should not crawl. For example, Disallow: /admin blocks the /admin section from crawling.

What does Allow do?

Allow explicitly permits crawling of a path. It is used when you want to open part of a section that is otherwise blocked by a Disallow rule.

What does Disallow: / mean?

Disallow: / blocks the entire website from crawling. This is useful for staging environments but dangerous for production sites if you want search engine indexing.

Why add a Sitemap to robots.txt?

A Sitemap entry helps search engines find your XML sitemap. The value is typically an absolute URL such as https://example.com/sitemap.xml.

Does robots.txt protect private pages?

No. robots.txt is not an access control mechanism. Private pages must be protected by authorization, server-side rules, and other security mechanisms.

What is the Host directive in robots.txt?

Host is a directive supported by some search engines and historically associated with Yandex. For most sites, correct Sitemap entries, canonicals, and redirects are more important.

Can robots.txt remove a site from search index?

You can add Disallow: /, but this prevents crawling rather than guaranteeing removal from the index. For index management, also use meta robots, HTTP headers, and webmaster tools.

Is my data sent to the server?

No. The robots.txt file is generated locally in the browser and is not sent to the server.

Do I need to register?

No. The tool works immediately without registration.

How to use

Choose a scenario: allow all crawling, block service sections, block the entire site, or configure rules manually.

Set User-agent, Allow, Disallow, and Sitemap URL.

Review any warnings, especially for Disallow: / or when blocking important site sections.

Copy or download robots.txt and place it in the root of your website.

More about this tool

All processing is done locally in the browser. Your data is not sent to the server.

Runs in the browser
Data is not sent to the server
Generates a robots.txt file
Supports User-agent, Allow, and Disallow
Adds Sitemap
Includes presets for common configurations
Shows warnings for risky rules
Can download the finished robots.txt
No registration required

Frequently Asked Questions

What is robots.txt?

robots.txt is a text file in the root of a website that tells search engine crawlers which sections may or may not be crawled.

Where should robots.txt be placed?

The file is typically placed in the website root, for example at https://example.com/robots.txt.

What does User-agent: * mean?

User-agent: * means the rules apply to all crawlers that honor robots.txt.

What does Disallow do?

Disallow specifies a path that crawlers should not crawl. For example, Disallow: /admin blocks the /admin section from crawling.

What does Allow do?

Allow explicitly permits crawling of a path. It is used when you want to open part of a section that is otherwise blocked by a Disallow rule.

What does Disallow: / mean?

Disallow: / blocks the entire website from crawling. This is useful for staging environments but dangerous for production sites if you want search engine indexing.

Why add a Sitemap to robots.txt?

A Sitemap entry helps search engines find your XML sitemap. The value is typically an absolute URL such as https://example.com/sitemap.xml.

Does robots.txt protect private pages?

No. robots.txt is not an access control mechanism. Private pages must be protected by authorization, server-side rules, and other security mechanisms.

What is the Host directive in robots.txt?

Host is a directive supported by some search engines and historically associated with Yandex. For most sites, correct Sitemap entries, canonicals, and redirects are more important.

Can robots.txt remove a site from search index?

You can add Disallow: /, but this prevents crawling rather than guaranteeing removal from the index. For index management, also use meta robots, HTTP headers, and webmaster tools.

Is my data sent to the server?

No. The robots.txt file is generated locally in the browser and is not sent to the server.

Do I need to register?

No. The tool works immediately without registration.