Wirebook › Lesson 17 of 18
What are bots doing on the web?
Right now, millions of little programs are reading the web, page after page, all day and all night. They're called bots (short for robots), and nearly half of all web traffic comes from them. Some are helpers: search-engine crawlers that read pages so you can find them later. Some are pests. Here you get to be a crawler on a tiny pretend web, then play bouncer and spot which visitors are bots.
What a crawler really does
- Start somewhere. A crawler begins with a list of addresses it already knows.
- Read the sign on the door. Before anything else, a polite crawler asks for
/robots.txt, a little text file where the site says which rooms robots should stay out of. Here's this site's real one: it asks robots not to poke at/api/. - Read the page and write it down. It notes the words, the title and where every link goes. Those notes become the search engine's index.
- Follow the links. Every new link goes on the to-do list, and round it goes. That's why a page nobody links to is so hard to find, unless the site lists it in a
sitemap.xml. - Don't stampede. A good crawler leaves a gap between visits to the same site, so it never makes the website tired.
robots.txt only works because good robots choose to obey it. A rude bot can walk straight past. So anything truly private needs a real lock, like a password, not just a "please keep out" sign. Your robot's diary shows exactly that. To try a real robots.txt against any path, use Tooltern's free robots.txt tester (paste the file — it never fetches).Good bots, pest bots
Helpers read the web so search engines can answer you, check that websites are still up (like this site's status page), translate pages and archive old ones so history isn't lost. They usually say who they are and obey the sign.
Pests copy whole websites, grab every concert ticket in a second, try thousands of passwords, or post junk in comment boxes. They hide who they are and go as fast as they can. Websites catch them with the same tricks you just used: things no person would do, like reading 300 pages in two seconds, filling in a box people can't even see (a honeypot trap), or ignoring the sign. Then they slow them down with a rate limit or show a puzzle that's easy for people and hard for bots.