Uploaded October 2016 | Updated September 2026, 2 hours ago
The goal of crawling the web and analyzing page contents is to construct a search index. An search index is not altogether different than the index of a book. Book indexes map key terms to pages on which those terms appear or are discussed. Search indexes map search terms to web pages on which those terms appear. However, multiple important differences distinguish book indexes from web search indexes. First, the web is constantly changing, so indexes must be automatically created and updated. Second, search engines must be prepared for users to search for anything and build indexes accordingly.
Credits: Talking: Geoffrey Challen (Assistant Professor, Computer Science and Engineering, University at Buffalo). Producing: Greg Bunyea (Undergraduate, Computer Science and Engineering, University at Buffalo).
Part of the internet-class.org online internet course. A blue Systems Research Group (https://blue.cse.buffalo.edu) production.
The goal of crawling the web and analyzing page contents is to construct a search index. An search index is not altogether different than the index of a book. Book indexes map key terms to pages on which those terms appear or are discussed. Search indexes map search terms to web pages on which those terms appear. However, multiple important differences distinguish book indexes from web search indexes. First, the web is constantly changing, so indexes must be automatically created and updated. Second, search engines must be prepared for users to search for anything and build indexes accordingly.
Credits: Talking: Geoffrey Challen (Assistant Professor, Computer Science and Engineering, University at Buffalo). Producing: Greg Bunyea (Undergraduate, Computer Science and Engineering, University at Buffalo).
Part of the internet-class.org online internet course. A blue Systems Research Group (https://blue.cse.buffalo.edu) production.
