Robots.txt

robots.txt

robots.txt is a simple text file placed at the root of a website to give guidance to automated crawlers about which parts of the site they may access. It follows a long-standing convention that many well-behaved crawlers honor when deciding what to visit and index. The file uses plain rules like "User-agent" to name a crawler and "Disallow" to list paths that should not be fetched. Because it is a voluntary standard, it does not technically enforce access, and malicious or poorly designed programs can ignore it. It also cannot hide content from the public internet because anyone who knows a URL can still load it in a browser. robots.txt matters because it helps site owners reduce server load, keep sensitive or duplicate pages out of search results, and guide respectful data collectors. For stronger protection, sites use authentication, noindex meta directives, or legal agreements to control access and reuse of content. Checking and updating this file is a simple way to improve site privacy and search engine behavior without changing website code. Understanding its limits and combining it with other tools gives site managers practical control over automated access.