Robots.txt Generator
Choose what search engines may crawl, list any folders to keep them out of, and add your sitemap address. Save the result as robots.txt at the top level of your site.
Fill in the fields above and the result shows here as you type.
Good to know
- robots.txt is a note on the front door for crawlers. It asks them not to fetch certain addresses. It is not a lock: anything private needs a password.
- Blocking a page here does not remove it from Google. To keep a page out of search results, leave it crawlable and use a noindex tag instead.
- AI crawlers are a choice. Blocking them keeps your text out of AI training; allowing them is how an AI assistant can name your business when someone asks.
This tool runs in your browser. What you type is not sent anywhere unless you ask for your report by email.
Get every result in one email.
Each tool you use adds its result to this list. When you are done, enter your email and I will send the lot as one report you can keep, forward or hand to whoever looks after your website.
Your results stay in this browser until you press send. The report is sent once. Your email goes to me and nowhere else. Privacy
Nothing here yet. Use any tool and its result is added automatically.
Rather have it done for you?
These tools show you what to fix. If you would rather hand it over, tell me what is not working and I will send back a working piece of the fix.