Generate a robots.txt file for your website with a visual builder. Control which pages search engines can crawl.
robots.txt is a text file that tells search engine crawlers which pages or sections of your site should not be crawled or indexed.
The robots.txt file must be placed in the root directory of your website (e.g., https://example.com/robots.txt).
No, robots.txt is a directive, not a rule. Well-behaved bots will follow it, but it doesn't prevent indexing. Use noindex meta tags for guaranteed de-indexing.