How do you find everything on the web? @internet-class

Uploaded October 2016 | Updated September 2026, 2 hours ago

The world wide web consists of billions of web pages and is growing and changing constantly. So how do you find all of those pages? The process typically utilizes an automated computer program known as a web crawler, which exploits the linked structure of the WWW to use one page to find more pages.

Credits: Talking: Geoffrey Challen (Assistant Professor, Computer Science and Engineering, University at Buffalo). Producing: Greg Bunyea (Undergraduate, Computer Science and Engineering, University at Buffalo).

Part of the internet-class.org online internet course. A blue Systems Research Group (https://blue.cse.buffalo.edu) production.

What are humans good at that computers are not?

How are domain names hierarchically organized?

What can and do websites track about you?

How do two parties exchange keys to communicate securely?

$Overview of a single internet search. Let’s review everything that has to happen for you to perform a single internet search. First, all of the billions of web pages have to be found and processed. That processing has to prepare the search engine to reply to any possible search. Second, the relative importance of various pages has to be established. Search engines use many different factors to determine how to rank results, but at the most basic level some pages are better than others—more reputable or more useful. Finally, the search engine has to receive your search, locate all relevant pages, rank them, and return the results. Amazingly, this entire process happens in a fraction of a second. Credits: Talking: Geoffrey Challen (Assistant Professor, Computer Science and Engineering, University at Buffalo). Producing: Greg Bunyea (Undergraduate, Computer Science and Engineering, University at Buffalo). Part of the https://www.internet-class.org online internet course. A blue Systems Research Group (https://blue.cse.buffalo.edu) production.$