Abstract:
Time-biased gain provides a unifying framework for information retrieval
evaluation, generalizing many traditional effectiveness measures while
accommodating aspects of user behavior not captured by these measures.
By using time as a basis for calibration against actual user data, time-biased
gain can reflect aspects of the search process that directly impact user
experience, including document length, near-duplicate documents, and summaries.
Unlike traditional measures, which must be arbitrarily normalized for averaging
purposes, time-biased gain is reported in meaningful units, such as the
total number of relevant documents seen by the user. In work reported at
SIGIR 2012, we proposed and validated a closed-form equation for estimating
time-biased gain, explored its properties, and compared it to standard
approaches. In work reported at CIKM 2012, we used stochastic simulation
to numerically approximate time-biased gain, an approach that provides
greater flexibility, allowing us to accommodate different types of user
behavior and increases the realism of the effectiveness measure. In work
reported at HCIR 2012, we extended our stochastic simulation to model the
variation between users. In this talk, I will provide an overview of time-biased
gain, and outline our ongoing and future work, including extensions to
evaluate query suggestion, diversity, and whole-page relevance. This is
joint work with Mark Smucker.
Biography:
Charles Clarke is a Professor in the School of Computer Science at the University of Waterloo, Canada. His research interests include information retrieval, web search, and text data mining. He has published on a wide range of topics, including papers related to question answering, XML, filesystem search, user interfaces, computational advertising, statistical natural language processing, and the evaluation of information retrieval systems. He is a co-author of the book Information Retrieval: Implementing and Evaluating Search Engines, MIT Press, 2010. He is co-Editor-in-Chief of the journal Information Retrieval. He is co-coordinator of the Web and Contextual Suggestion Tracks at TREC. He frequently serves on the program committees of information retrieval conferences, including SIGIR, CIKM, ECIR, and WSDM.
