INLS490-154W: Information Retrieval Systems Design and Implementation http://www.inforetrieval.org/2009_fall/inls490_154w/ en-us Copyright 2009 Chirag Shah Fall 2009 web-based course Chirag Shah Tools for organizing and accessing information have become indispensable. It is critical, therefore, to understand their design and operational foundations. In this course students will have an opportunity to learn about search engines, web crawling, and some Web 2.0 technologies based on hands-on experience and with a focus on techniques that can be used to access, retrieve, organize, and present information. Students will work with practical developmental tools and learn relevant concepts through experimentation. For instance, students will employ an open source search engine and learn about indexing, retrieving, and ranking techniques. Tools for organizing and accessing information have become indispensable. It is critical, therefore, to understand their design and operational foundations. In this course students will have an opportunity to learn about search engines, web crawling, and some Web 2.0 technologies based on hands-on experience and with a focus on techniques that can be used to access, retrieve, organize, and present information. Students will work with practical developmental tools and learn relevant concepts through experimentation. For instance, students will employ an open source search engine and learn about indexing, retrieving, and ranking techniques. Copyright 2009 Chirag Shah information,retrieval,IR,systems,design,implementation,online,course no information,retrieval,IR,systems,design,implementation,online,course Education/Training Technology/Software How-To http://creativecommons.org/licenses/by-nc-nd/2.0/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/inls490-154w.rss http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/logo_small.jpg Information Retrieval Systems Design and Implementation chirag@unc.eduChirag Shah Welcome to the Course Chirag Shah What’s this course all about This is a pre-course podcast welcoming the students to the course and providing some information about how this course will work. Sat, 15 Aug 2009 19:00:00 EST 4:40 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/7miOyn-Pv7A/ http://www.inforetrieval.org/2009_fall/inls490_154w/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-0.m4v chirag@unc.edu (Chirag Shah) Introduction Chirag Shah Get ready, get started, get going This is the podcast for the first class, in which the course and its structure is introduced. MySQL is introduced with some working examples. Fri, 28 Aug 2009 10:00:00 EST 34:21 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/7miOyn-Pv7A/ http://www.inforetrieval.org/2009_fall/inls490_154w/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-1.m4v chirag@unc.edu (Chirag Shah) IR with MySQL and Text Files Chirag Shah Moving from structured to unstructured IR This is the podcast for the second class, in which we continue working with MySQL and look at full-text search in database, create a basic UI for database access, and start looking at IR from unstructured data sources. Tue, 01 Sep 2009 10:00:00 EST 1:09:01 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-2.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/7miOyn-Pv7A/ chirag@unc.edu (Chirag Shah) Learning to index Chirag Shah Better representation for better retrieval This is the podcast for the third class, in which we go through indexing process by first doing tokenization, stop words removal, and stemming manually, and then with the help of Lemur Toolkit. Tue, 08 Sep 2009 10:00:00 EST 1:29:18 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-3.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) Query processing and retrieval Chirag Shah Mapping information need to information This is the podcast for the fourth class, in which we look at how queries can be processed using Lemur Toolkit. We then match the processed queries with collection index to retrieve a rank list. Tue, 15 Sep 2009 10:00:00 EST 58:05 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-4.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) Retrieval models-1 Chirag Shah Relevant or not relevant that is the question This is the podcast for the fifth class, in which we dive into models for retrieving information. We had already looked at the vector space model. Now we talk about boolean retrieval and language models for retrieval. Tue, 22 Sep 2009 10:00:00 EST 55:05 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-5.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) Retrieval models-2 Chirag Shah Relevant or not relevant that is the question This is the podcast for the sixth class, in which we continue exploring models for retrieving information. So far we have seen vector space, boolean, and language models. Now we talk about probabilistic and relevance models. Tue, 29 Sep 2009 10:00:00 EST 1:07:32 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-6.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) Structured query processing Chirag Shah There’s more to a query than meets the eye This is the podcast for the seventh class, in which we look at how structured queries with term weights can be executed using Lemur. We use this to provide terminologic feedback to the user, and incorporate the relevance feedback that the user provides into retrieval process. Tue, 06 Oct 2009 10:00:00 EST 51:16 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-7.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) Evaluation-1 Chirag Shah Art of identifying good, bad and ugly This is the podcast for the eighth class, in which we start looking at one of the core components of IR - evaluation. We begin our discussion by revisiting recall and precision, and then continue exploring R-precision, AP, and MAP. We see how these can be measured manually and then using TREC supplied tools. Tue, 06 Oct 2009 11:00:00 EST 41:53 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-8.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) Evaluation-2 Chirag Shah Art of identifying good, bad and ugly This is the podcast for the ninth class, in which we continue looking at evaluation. We talk about more measures to evaluate a query and a system. We also look at comparing two rank lists. Tue, 13 Oct 2009 10:00:00 EST 51:25 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-9.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) User interface for search Chirag Shah Connecting systems to people This is the podcast for the tenth class, in which we connect the back-end for search that we have been working with to a web-based front-end. This is done using Indri, a new search engine component for Lemur. We also explore some details of AJAX and see how we could use it to enhance our user interface for search. Tue, 27 Oct 2009 10:00:00 EST 01:25:59 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-10.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) Web crawling Chirag Shah Crawl before you walk This is the podcast for the eleventh class, in which we see traditional and non-traditional methods of collecting data off the web. Traditional way is demonstrated using web crawling using wget, and non-traditional way is instantiated with YouTube harvesting. Tue, 03 Nov 2009 10:00:00 EST 58:47 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-11.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) IR on Web 2.0 Chirag Shah Some business, some buzz This is the podcast for the twelfth class, in which we see how REST requests can be made through the web, and the responses in XML can be parsed. This allows us to start connecting with Web 2.0 sources that provide the functionality of meshing different sources by open data exchange. Tue, 10 Nov 2009 10:00:00 EST 46:59 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-12.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) Information organization Chirag Shah Creating order from chaos This is the podcast for the thirteenth class, in which we look at a couple of ways to organize and present information to the user. We see how a term-cloud interface can be created, allowing the user to get a quick glance at the underlying collection. We also talk about a number of clustering algorithms and see how they can be implemented with Lemur. Tue, 17 Nov 2009 10:00:00 EST 49:15 information, retrieval, IR, systems, design, implementation, online course no <img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/Ut678ndDhbE" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/7miOyn-Pv7A" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/gYfB1dpRyeA" height="1" width="1"/><img src="http://feeds.feedburner.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~4/QRT04d6sUB8" height="1" width="1"/> http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/QRT04d6sUB8/ http://www.inforetrieval.org/2009_fall/inls490_154w/podcasts/INLS%20490-154W%20Podcast-13.m4v http://feedproxy.google.com/~r/Inls490-154wInformationRetrievalSystemsDesignAndImplementation/~3/gYfB1dpRyeA/ chirag@unc.edu (Chirag Shah) Chirag Shah nonadult Fall 2009 web-based course nonadult