robots.txt maker

Choose what search engines may visit and get a robots.txt file ready to upload.

Runs in your browser. Nothing is uploaded.

How to use

  1. Choose whether search engines may visit the site.
  2. List any paths to keep them out of, one per line, such as /admin/.
  3. Add your sitemap address.
  4. Copy the result into a file called robots.txt at the top of your site.

Good to know

  • robots.txt asks well-behaved crawlers to stay out. It does not hide or protect pages.
  • To keep a page out of Google results, use a noindex tag instead.

Questions

Where does robots.txt go?
At the root of the site, for example example.com/robots.txt.
Will blocking a page remove it from Google?
Not always. Google can still list a blocked address if other sites link to it. Use noindex.
Should I include my sitemap?
Yes. It helps search engines find all your pages.

Also known as: robots.txt generator, create robots.txt, robots file maker.

More website tools