Robots.txt Generator

Online robots.txt generator with User-agent, Allow, Disallow, and Sitemap for basic crawl settings.

Обработка выполняется локально в браузере. Данные не отправляются на сервер.

For this tool, processing runs locally in your browser and data is not sent to the server.

Robots.txt Generator is a free online tool for creating a starter robots.txt file for your site. Processing runs locally in your browser.

What it does

Output

Processing

Continue working

Choose a related tool for the next step.

Data passed between tools is kept only in the current browser session.

How to use

Choose a scenario: allow indexing, block service sections, block the whole site, or configure rules manually.
Specify User-agent, Allow, Disallow, and Sitemap URL.
Review warnings, especially for Disallow: / or blocking important site sections.
Copy or download robots.txt and place it in the site root.

More about this tool

Robots.txt Generator helps create a starter robots.txt for a website with User-agent, Allow, Disallow, Sitemap, and optionally Host.

Suitable for new sites, test environments, SEO setup, blocking service sections, and draft preparation before publishing.

robots.txt gives instructions to crawlers but does not protect private pages and does not replace authentication, server access controls, meta robots, canonical tags, or a full SEO audit.

Processing runs locally in your browser.

Generates robots.txt
Supports User-agent, Allow, and Disallow
Adds Sitemap
Has scenarios for common setups
Shows warnings for risky rules
Can download ready robots.txt
Runs in the browser
Data is not sent to the server
No registration required

Frequently Asked Questions

What is robots.txt?

robots.txt is a text file in the site root that tells search crawlers which sections may or may not be crawled.

Where should robots.txt be placed?

Usually in the site root, e.g. https://example.com/robots.txt.

What does User-agent: * mean?

User-agent: * means the rules apply to all crawlers that respect robots.txt.

What does Disallow do?

Disallow specifies a path crawlers should not crawl. For example, Disallow: /admin blocks the /admin section from crawling.

What does Allow do?

Allow explicitly permits crawling of a path, often to open part of a section otherwise blocked by Disallow.

What does Disallow: / mean?

Disallow: / blocks the entire site from crawling. Useful for test environments but risky for production if you want indexing.

Why add Sitemap to robots.txt?

Sitemap helps search engines find the XML sitemap. Usually an absolute URL such as https://example.com/sitemap.xml is used.

Does robots.txt protect private pages?

No. robots.txt is not access control. Protect private pages with authentication, server rules, and other security mechanisms.

What is Host in robots.txt?

Host is not supported by all search engines and was more common in Yandex context. For most sites Sitemap, canonical, and redirects matter more.

Can I block indexing with robots.txt?

You can add Disallow: /, but that blocks crawling, not always removal from the index. Also use meta robots, headers, and webmaster tools.

Is my data sent to the server?

No. Processing runs locally in your browser and data is not sent to the server.

Do I need to register?

No. The tool works without registration.

How to use

Choose a scenario: allow indexing, block service sections, block the whole site, or configure rules manually.

Specify User-agent, Allow, Disallow, and Sitemap URL.

Review warnings, especially for Disallow: / or blocking important site sections.

Copy or download robots.txt and place it in the site root.

More about this tool

Robots.txt Generator helps create a starter robots.txt for a website with User-agent, Allow, Disallow, Sitemap, and optionally Host.

Suitable for new sites, test environments, SEO setup, blocking service sections, and draft preparation before publishing.

robots.txt gives instructions to crawlers but does not protect private pages and does not replace authentication, server access controls, meta robots, canonical tags, or a full SEO audit.

Processing runs locally in your browser.

Generates robots.txt
Supports User-agent, Allow, and Disallow
Adds Sitemap
Has scenarios for common setups
Shows warnings for risky rules
Can download ready robots.txt
Runs in the browser
Data is not sent to the server
No registration required

Frequently Asked Questions

What is robots.txt?

robots.txt is a text file in the site root that tells search crawlers which sections may or may not be crawled.

Where should robots.txt be placed?

Usually in the site root, e.g. https://example.com/robots.txt.

What does User-agent: * mean?

User-agent: * means the rules apply to all crawlers that respect robots.txt.

What does Disallow do?

Disallow specifies a path crawlers should not crawl. For example, Disallow: /admin blocks the /admin section from crawling.

What does Allow do?

Allow explicitly permits crawling of a path, often to open part of a section otherwise blocked by Disallow.

What does Disallow: / mean?

Disallow: / blocks the entire site from crawling. Useful for test environments but risky for production if you want indexing.

Why add Sitemap to robots.txt?

Sitemap helps search engines find the XML sitemap. Usually an absolute URL such as https://example.com/sitemap.xml is used.

Does robots.txt protect private pages?

No. robots.txt is not access control. Protect private pages with authentication, server rules, and other security mechanisms.

What is Host in robots.txt?

Host is not supported by all search engines and was more common in Yandex context. For most sites Sitemap, canonical, and redirects matter more.

Can I block indexing with robots.txt?

You can add Disallow: /, but that blocks crawling, not always removal from the index. Also use meta robots, headers, and webmaster tools.

Is my data sent to the server?

No. Processing runs locally in your browser and data is not sent to the server.

Do I need to register?

No. The tool works without registration.