Question

When searching the web software programs called fetch a few web pages and then they follow the links on those pages and fetch the pages they point to, and so forth?

Answer 1

Software programs called Spiders

When you are on Google and you decide to do a search, you are not searching the web pages. What you are doing is actually searching Google’s index of the web page. This is done with programs called spiders. They start off by fetching a few web pages, then follow links to those pages and collect all the pages they point to; and follow all the links, and then again fetch the pages they link to, and so forth. This is done until we have indexed a huge chunk of the web.