COMP 475 - Search Engine Development

Dr. Frank McCown
Science 206
TR 3:00 - 4:15 pm

Class links

Syllabus
Blackboard - For taking quizes
Easel - All homework and projects are here

Textbooks

Required book: An Introduction to Search Engines and Web Navigation by Levene
Optional book: Google's PageRank and Beyond: The Science of Search Engine Rankings by Langville and Meyer
Online text: An introduction to information retrieval by Manning, Raghavan, and Schütze

Search Engines

Introduction to Search Engines (PowerPoint slides)
Web Search Engine Wikipedia article
List of search engines
Popularity: Market share of US searches in 2007
A Comparison of Open Source Search Engines by Middleton and Baeza-Yates (2007) and rebuttal
Compare Google and Yahoo results

Tomcat, Java, and Java servlets

Nutch

Web Chracterization

Map of the Internet
W3C Web Characterization Terminology & Definitions Sheet
Trends in the Evolution of the Public Web: 1998-2002 by O'Neill, Lavoie, Bennett (2003)
Graph Structure in the Web by Broder et al. (2000)
The Indexable Web is more than 11.5 billion pages by Gulli and Signorini (2005)
Characterization of national Web domains by Baeza-Yates, Castillo, and Efthimiadis

Search Queries

Web search query Wikipedia article
Google Trends
Searching the web: The public and their queries by Spink, Wolfram, Jansen, Saracevic (2000)
AOL's disturbing glimpse into users' lives by McCullagh (2006)
History Repeats Itself: Repeat Queries in Yahoo’s Logs by Teevan, Adar, Jones, Potts (2006)

Web Crawling

Web Crawler Wikipedia article
Simple web crawler written in Java
Heritrix web crawler
Web Robots Pages - Robots Exclusion Protocol
Academic Papers

Site Design Impact on Robots by Smith and Nelson (2008)
Mercator: A Scalable, Extensible Web Crawler by Heydon and Najork (1999)
Crawling the Web by Pant, Srinivasan, and Menczer (2004)
RankMass Crawler: A Crawler with High Personalized PageRank Coverage Guarantee by Cho and Schonfeld (2007)
IRLbot: Scaling to 6 Billion Pages and Beyond by Lee, Leonard, Wang, and Loguinov (2008)

Sitemaps

Sitemaps: Above and Beyond the Crawl of Duty by Schonfeld & Shivakumar (2009)

Indexing

Search engine indexing Wikipedia article
Inverted Index Wikipedia article
TF-IDF (term frequency–inverse document frequency) Wikipedia article
Ch 1: Boolean retrieval
Ch 2: The term vocabulary and postings lists
Document Indexing Tutorial
Porter Stemming Algorithm and various implementations
Inverted files for text search engines by Zobel and Moffat (2006)

Ranking

PageRank Wikipedia article
The PageRank Citation Ranking: Bringing Order to the Web by Page, Brin, Motwani, Winograd (1998)
HITS Algorithm Wikipedia article
Authoritative Sources in a Hyperlinked Environment by Kleinberg (1999)
Google eye-tracking study
Google page ranking blog posts (2008): 1 2 3

Web Spam

Spamdexing Wikipedia article
Adversarial IR on the Web (PowerPoint slides)
Matt Cutts (Google) discussing web spam
Combating Web Spam with TrustRank by Gyongyi, Garcia-Molina, and Pedersen (2004)

Finding Duplicates

Detecting Near Duplicates for Web Crawling by Manku, Jain, and Sarma (2007)
Syntactic Clustering of the Web by Broder, Glassman, Manasse, and Zweig (1997)
Finding Near-Duplicate Web Pages: A Large-Scale Evaluation of Algorithms by Henzinger (2006)

Other Search Engine Courses

WWW Search Engines at Lehigh University
Web Search Engines at New York University
Information Retrieval and Web Search at University of North Texas
Information Retrieval and Web Search at University of Texas at Austin
Information Retrieval and Search Engines at Penn State
Search Engine Technologies at University of Texas at El Paso
Search Engines and Web Navigation at Birkbeck University of London

Page last modified: