What is a web crawler?
In one sentence: A bot that reads web pages one after another by following their links, so a search engine can note what each page says.
A little more
A crawler (also called a spider) starts with a few web addresses it knows. It fetches each page, writes down its words and title, and adds every link it finds to a to-do list. Then it fetches the next one, again and again, all day.
Its notes become a search engine's index: for each word, the pages it appears on. That's why a search can answer in a split second: it looks in the notebook instead of reading the whole web again. A page that nothing links to is hard for a crawler to find unless the site lists it in a sitemap.
It's a bit like… A librarian who walks through every room, follows every sign to the next room, and writes each book's topics on a card.