NTCIR

Evaluation measures are the basis for quantifying the performance of IR systems and the way in which their values can be processed to perform statistical analyses depends on the scales on which these measures are defined. For example, mean and variance should be computed only when relying on interval scales.
In this talk, we will present our formal theory of IR evaluation measures, based on the representational theory of measurements, to determine whether and when IR measures are interval scales. We found that common set- based retrieval measures – namely Precision, Recall, and F-measure – always are interval scales in the case of binary relevance while this does not happen in the multi-graded relevance case. In the case of rank-based retrieval measures – namely AP, gRBP, DCG, and ERR – only gRBP is an interval scale when we choose a specific value of the parameter p and define a specific total order among systems while all the other IR measures are not interval scales. We will also introduce some brand new set-based and rank-based IR evaluation measures which ensure to be interval scales.
In our previous work we defined a theory of IR evaluation measures, based on the representational theory of measurement, which allowed us to determine whether and when IR measures are interval scales.
Finally, we will discuss the outcomes of an extensive evaluation, based on standard TREC collections, to study how our theoretical findings impact on the experimental ones. In particular, we conduct a correlation analysis to study the relationship among the above-mentioned state-of-the-art evaluation measures and their scales. We study how the scales of evaluation measures impact on non parametric and parametric statistical tests for multiple comparisons of IR system performance.

Nicola Ferro is associate professor in computer science at the University of Padua, Italy. His research interests include information retrieval, its experimental evaluation, multilingual information access and digital libraries. He is the coordinator of the CLEF evaluation initiative, which involves more than 200 research groups world-wide in large-scale IR evaluation activities. He was the coordinator of the EU Seventh Framework Programme Network of Excellence PROMISE on information retrieval evaluation. He is associate editor of ACM TOIS and was general chair of ECIR 2016.

MediaEval is a multimedia benchmarking initiative which seeks to evaluate new algorithms for multimedia access and retrieval. MediaEval emphasizes the "multi" in multimedia, including tasks combining various facet combinations of speech, audio, visual content, tags, users, and context. MediaEval innovates new tasks and techniques focusing on the human and social aspects of multimedia content in a community driven setting. The initiative provides a platform for researchers to organize benchmark tasks within a planned annual timeline and to report results at an end of campaign workshop. MediaEval 2019 marks the 10th anniversary of the foundation of MediaEval. This presentation will briefly overview 10 years of MediaEval campaigns and summarize current activities within the MediaEval 2019 campaign.

Gareth Jones conducts research on multiple topics in information retrieval, including multimedia, multilingual and personal content across a wide range of application areas. Over the last 20 years he has published hundreds of papers describing this work at multiple venues. Much of his research encompasses the design of tasks for the evaluation of this research, including test collections and evaluation metrics. Since 2002 he has been responsible for the organisation of international benchmarking tasks at venues including CLEF, FIRE, NTCIR and TRECVid. In 2010, together with Martha Larson, Delft University of Technology, The Netherlands, he co-founded the MediaEval Multimedia Benchmarking initiative to provide a platform for the development and evaluation of novel tasks in multimedia indexing and search. Gareth has served as co-Programme Chair for ECIR 2011, Information Retrieval Chair for ACM CIKM 2010, and co-Chair of ACM SIGIR 2013 and CLEF 2017 (with MediaEval 2017) both hosted in Dublin.
Gareth is a faculty member of the School of Computing, Dublin City University (DCU), Ireland and a Principal Researcher in the SFI ADAPT Centre. He holds B.Eng. and PhD degrees from the University of Bristol, UK. He has previously held posts at the University of Cambridge and University of Exeter, U.K., and in 1997 was a Toshiba Fellow at the Toshiba Corporation Research and Development Center in Kawasaki, Japan.