Robots.txt Generator

What is a robots.txt file?

A robots.txt file is a small text file in the root folder of your website, for example example.com/robots.txt. It tells crawlers such as Googlebot and Bingbot which parts of your site they may visit. It is the first file most bots read when they arrive.

Robots.txt is used to:

• keep crawlers out of admin areas, carts, internal search results and other low-value pages
• point bots to your XML sitemap
• allow or block specific bots, including AI training crawlers such as GPTBot and Google-Extended
• slow down aggressive crawlers with a crawl delay (Bing and Yandex respect it, Google ignores it)

How to create your robots.txt

1. Choose whether all robots may crawl the site. For almost every live website, keep Allow.
2. Add your sitemap URL, for example https://example.com/sitemap.xml.
3. List folders or pages to block, one per line, such as /wp-admin/ or /cart/.
4. Add exceptions if a file inside a blocked folder must stay open, like /wp-admin/admin-ajax.php.
5. Optionally set rules for individual bots.
6. Click Download robots.txt and upload the file to the root folder of your site.

Important: robots.txt does not hide pages

This is the most common misunderstanding. Blocking a page in robots.txt stops crawling, not indexing. If other sites link to a blocked page, Google can still list its URL without a description. To keep a page out of search results, let Google crawl it and add a noindex robots meta tag instead, which you can create with our Meta Tag Generator.

Also, robots.txt is public. Anyone can read it, so never list secret folders there. Protect private areas with a password.

Google's introduction to robots.txt covers the rules in detail.

Robots.txt mistakes to avoid

• Disallow: / under User-agent: * blocks your entire site. It is often left behind after a website goes live.
• Blocking CSS and JavaScript files can stop Google from rendering your pages properly.
• Paths are case-sensitive: /Blog/ and /blog/ are different.

After uploading, run a Website SEO Audit, which confirms that robots.txt and sitemap.xml are reachable. If you do not have a sitemap yet, build one with the XML Sitemap Generator.

Robots.txt syntax explained

A robots.txt file is a plain text file made of groups. Each group starts with a User-agent line, followed by rules:

Paths are case sensitive and start from the root of the domain. An empty Disallow: line means nothing is blocked. Google's robots.txt specification is the official reference.

Ready-made examples

Allow everything (most small sites):

``` User-agent: * Disallow:

Sitemap: https://example.com/sitemap.xml ```

Typical WordPress site:

``` User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml ```

Block one folder for all bots:

`` User-agent: * Disallow: /private-files/ ``

Should you block AI crawlers?

Several AI companies publish the names of their crawlers so site owners can opt out, for example GPTBot (OpenAI), Google-Extended (Google's AI training control), CCBot (Common Crawl) and ClaudeBot (Anthropic). Blocking them may keep your content out of some AI training data, but it may also mean your brand appears less in AI answers. There is no single right answer; decide based on whether you want visibility or control. This generator has one-click options for GPTBot, Google-Extended and CCBot.

Testing your robots.txt

After uploading the file to the root of your domain, open https://yourdomain.com/robots.txt in a browser to confirm it is live. Then use the robots.txt report in Google Search Console to see which version Google fetched and whether it found errors. If a page is unexpectedly missing from Google, check it with the Meta Tag Checker as well, because a noindex tag is a more common cause than robots.txt.

Always pair your robots.txt with a good sitemap. If you do not have one yet, create it with the XML Sitemap Generator and add its URL to the Sitemap: line.

Example: a small business website

A typical small business site with a contact form and a private admin area might use:

``` User-agent: * Disallow: /admin/ Disallow: /thank-you/

Sitemap: https://example.com/sitemap.xml ```

This lets every search engine crawl the public pages, keeps the admin area and the form confirmation page out of crawling, and points bots to the sitemap. Remember that the thank-you page should also carry a noindex tag if you do not want it in search results.

Quick robots.txt checklist

• The file is called robots.txt, in lower case, at the root of the domain.
• It is plain text, saved as UTF-8.
• Each subdomain has its own file if needed, for example shop.example.com/robots.txt.
• CSS, JavaScript and image folders are not blocked.
• The Sitemap line uses the full https address.
• After changes, the file was tested in Google Search Console's robots.txt report.

Last updated: October 25, 2026