This is a critical distinction. Disallow in a robots.txt file tells a bot not to crawl a page. The page can still appear in search results. The noindex directive (an HTML meta tag) tells a bot not to index a page, which prevents it from appearing in search results. For a page to be truly hidden, you need to use noindex.

No. The robots.txt file is a publicly viewable document. Anyone can type yourwebsite.com/robots.txt into their browser. Therefore, it should never be used to hide sensitive information. Other methods like password protection or noindex are more effective for securing private content.

The most common mistakes include using the wrong case, accidentally disallowing the entire site with Disallow: /, or disallowing essential resources like CSS and JavaScript files, which can prevent a site from rendering correctly for a search engine.

The robots.txt file itself is tiny and has no direct effect on your site’s speed for users. However, by helping to manage crawler activity, it can reduce server load and help with overall website performance.

No. Each subdomain, such as blog.example.com, needs its own separate robots.txt file in its root directory.