Robots.txt Generator

What is a robots.txt file?

A robots.txt file is a small text file at the root of your website that tells search engine crawlers which parts of the site they may visit. Before a well-behaved bot such as Googlebot crawls your pages, it looks for this file at yourwebsite.com/robots.txt and reads the rules.

The rules follow the Robots Exclusion Protocol, an old web standard that was formally published as RFC 9309 in 2022. A robots.txt file does not ban visitors and it is not a security tool. It is a polite request that reputable crawlers follow, mainly to save crawl time on pages that do not matter.

The main robots.txt rules

• User-agent: Names the crawler that the rules below it apply to. An asterisk means every crawler.
• Disallow: A path the crawler should not visit. Disallow: /admin/ blocks the admin folder, and an empty Disallow allows everything.
• Allow: A path that is allowed even inside a blocked folder, such as the AJAX file inside the blocked WordPress admin area.
• Sitemap: The full address of your XML sitemap, which helps crawlers find your pages.
• Crawl-delay: Asks a bot to wait a number of seconds between requests. Bing respects it, but Google ignores it.

Where to put your robots.txt file

The file must be named robots.txt, saved as plain UTF-8 text and placed in the root folder of your domain, so that it opens at https://yourwebsite.com/robots.txt. A file inside a sub folder is ignored. Each subdomain, such as blog.yourwebsite.com, needs its own file.

On WordPress, Shopify and many website builders you edit the file through a setting or a plugin. In Next.js you can create a robots file in the app folder, and on a normal host you upload it with the file manager or FTP.

Example robots.txt files

To allow every crawler to visit your whole site, use User-agent: * followed by an empty Disallow line. This is the best choice for most websites. To block the entire site, which suits a staging or private site, use User-agent: * followed by Disallow: /.

A typical WordPress file blocks /wp-admin/, allows /wp-admin/admin-ajax.php and adds the sitemap line at the end. An online shop often blocks the cart, checkout and account pages, because they have no value in search results. The presets above create these files in one click.

Wildcards and which rule wins

Google and Bing understand two special characters. The star matches any run of characters, so Disallow: /*?sessionid= blocks every address with that parameter. The dollar sign marks the end of an address, so Disallow: /*.pdf$ blocks only addresses that end in .pdf.

When rules overlap, Google follows the most specific one, which is the longest matching path. If an Allow and a Disallow match equally, the Allow wins. Paths are case-sensitive, so /Photos/ and /photos/ are different folders.

Robots.txt is not the same as noindex

Blocking a page in robots.txt stops crawlers from reading it, but the address can still appear in search results, without a description, if other sites link to it. To keep a page out of search results, add a noindex meta tag or an X-Robots-Tag header, and leave the page crawlable so the search engine can see that tag.

For private information, use a password and not robots.txt. The file itself is public, so anyone can read it, and listing a secret address there only advertises it.

Should you block AI crawlers?

Some companies run crawlers that collect pages to train AI models, such as GPTBot from OpenAI, ClaudeBot from Anthropic, CCBot from Common Crawl and PerplexityBot. You can ask them to stay away with a separate User-agent group and Disallow: /. Google-Extended is a special token that controls whether your content is used for Google's Gemini AI, and blocking it does not change how your site ranks in Google Search.

These are requests, not locks. Reputable companies follow them, but not every bot does. Whether to block them is a business decision, so tick the ones you want and leave the rest.

Common robots.txt mistakes

• Blocking the whole site by accident: A Disallow: / left over from a staging site can remove a live site from search results.
• Blocking CSS and JavaScript: Google needs these files to see your pages the way visitors do.
• Forgetting how matching works: Disallow: /blog also blocks /blog-news, because rules match the start of an address. Use /blog/ to block only the folder.
• Listing secret URLs: The file is public, so it shows everyone where the hidden pages are.
• Putting the file in the wrong place: It must sit in the root of the domain, not in a folder.
• Using noindex inside robots.txt: Google does not support it there. Use a meta tag instead.

How to test your robots.txt file

Open yourwebsite.com/robots.txt in a browser to check that it loads. Then use the robots.txt report in Google Search Console, which shows the version Google found and any errors. After you change the file, give crawlers some time, because Google can keep a copy of it for up to a day.

Why use our free robots.txt generator?

• One-click presets: Allow everything, block everything, WordPress and online shop.
• Safety warnings: Alerts for blocking the whole site, blocking CSS and JavaScript, and wrong sitemap links.
• AI crawler options: Block the main AI bots with a tick.
• Copy or download: Get the file as robots.txt, ready to upload.
• Private and free: Nothing is uploaded, and there is no signup or limit.