Wirebook › Glossary › robots.txt
What is robots.txt?
In one sentence: A small text file at the top of a website that tells robots which parts they should stay out of. Good bots obey it; it isn't a lock.
A little more
robots.txt lives at the very top of a website, for example example.com/robots.txt. Lines like Disallow: /private ask robots to keep out of those pages, and a Sitemap: line points them to a list of every page worth reading.
Polite crawlers, like the big search engines, always read it first and obey it. But it is only a request: a rude bot can ignore it. Anything truly private needs a real lock, such as a password, not just a sign.
It's a bit like… A 'staff only' sign on a door. Most people respect it, but it doesn't actually lock anything.