Create a robots.txt file that tells search engines what they may crawl. Block private folders, keep AI training crawlers out if you wish, add your sitemap, and check any address against the rules the way Google reads them.
loading toolHow to make a robots.txt file
- Choose whether search engines may crawl your site.
- List the folders to block, add your sitemap, and tick AI crawlers if you want them out.
- Test a few addresses, then copy or download robots.txt.
Good to know
- Google follows the most specific (longest) matching rule, so
Allow: /admin/public/beatsDisallow: /admin/. *matches anything and$marks the end, as inDisallow: /*.pdf$.
Questions
Does robots.txt hide a page from Google?
Not reliably. It stops crawling, but a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of search.
Where do I put the file?
In the main folder of your site, so it opens at https://yoursite.com/robots.txt. On Blogger: Settings → Crawlers and indexing → Custom robots.txt.
Will blocking AI crawlers affect Google Search?
No. Google-Extended only controls use of your content for Google’s AI models; normal Googlebot crawling for search continues.
Is my data private?
Yes. Everything happens inside your own browser: nothing you type or choose is uploaded to us or anyone else.
Is it free?
Completely free, with no sign-up, no watermark and no limit on how many times you use it. Light ads keep the site running.
