NTCIR-14 Conference


NTCIR-14 Conference Keynote

Date: June 10th (Tue), 2019 (Time: 10:10 a.m. - 11:10 a.m.)
Location: Hitotsubashi-hall, NII, Tokyo, Japan

Title: An Academic in a Data Wonderland: Five Lessons from Commercial Search

Speaker: Charles L. A. Clarke(University of Waterloo, Canada)

Prof. Charles L. A. Clarke


For over two years, I put my academic research career on hold, left the University of Waterloo, moved to California, and worked to make a large commercial search engine better. In September 2019, I returned to my role as a Professor at the University of Waterloo, but the things I learned in the commercial world will impact my academic research for the rest of my career. While I can't tell you deep, dark secrets, I can you about problems and ideas that are inspired by my experience. These include changes in the way I think about search evaluation, online search behaviour, personalization and query understanding.


Charlie Clarke is a Professor in the School of Computer Science at the University of Waterloo, Canada. His research interests span the broad field of information retrieval, including search, question answering, and recommendation. He has published over 100 refereed contributions on a wide range of topics, including information retrieval evaluation, user interfaces, filesystem search, natural language processing, machine learning, databases, and XML. He is an ACM Distinguished Scientist and Past Chair of the ACM Special Interest Group on Information Retrieval (SIGIR). He frequently serves on the program committees of major information retrieval conferences, including WSDM, WWW, SIGIR, and CIKM. He was Program Co-Chair for SIGIR in 2007 and 2014. He is a co-author of the book Information Retrieval: Implementing and Evaluating Search Engines, MIT Press, 2010. From August 2016 to August 2018, while on leave from Waterloo, he was an engineering tech lead at Facebook, where he worked on metrics and machine learning for Facebook Search.


EVIA 2019 Keynote

EVIA 2019 Keynote

Date: June 10th (Monday), 2019 (Time: 13:15 p.m. - 14:15 p.m.)
Location: Hitotsubashi Conference Room 2-4, NII, Tokyo, Japan

Title: How to turn the noise down in test collection measurement

Speaker: Mark Sanderson (RMIT University, Australia)

Prof. Mark Sunderson


There has long been a common view amongst Information Retrieval (IR) researchers on the main source of statistical noise when measuring the effectiveness of an IR system using a test collection. The common belief was that variation across topics was that source. Consequently, we commonly build collections with many topics in order to minimise the noise. In this talk I will outline two research projects involving RMIT researchers in collaboration with others in Melbourne, Padova, and Microsoft Research. The conclusions of these works challenge the common view. Noise is more complicated than previously thought; topic variation however, is not the problem. Substantial sources of noise exist in the document collections that we search and in the query variations we use. In this talk, I will describe the recent research and discuss the implications of the results.


Mark Sanderson is Professor of Information Retrieval at RMIT University where he is head of the RMIT Information Retrieval (IR) group. Mark received his Ph.D. in Computer Science from the University of Glasgow, United Kingdom, in 1997. He has raised over $10 million dollars in grant income, published hundreds of papers, and has over 9,000 citations to his work. He has 25 current and/or past PhD students. In collaboration with one student, Mark was the first show the value of snippets, a component of search engine interfaces which are now a standard feature of all engines. One of Mark's papers was given an honourable mention at SIGIR's 2017 test of time awards. Mark has been co-editor of Foundations and Trends in Information Retrieval; associate editor of IEEE TKDE, ACM TOIS, ACM TWeb, and IP&M; and served on the editorial boards of IRJ and JASIST. Mark was general chair of ACM SIGIR in 2004. He was a PC chair of ACM SIGIR 2009 & 2012; and ACM CIKM 2017. Prof Sanderson is also a visiting professor at NII in Tokyo.


NTCIR-14 Invited Talks

NTCIR-14 Invited Talk 1

Date: June 11th (Tuesday), 2019 (Time: 17:00 p.m. - 18:00 p.m.)
Location: Hitotsubashi-hall, NII, Tokyo, Japan

Title: Evaluation Measures Scales: From Theory to Experimentation

Speaker: Nicola Ferro (University of Padua, Italy)

Prof. Nicola Ferro


Evaluation measures are the basis for quantifying the performance of IR systems and the way in which their values can be processed to perform statistical analyses depends on the scales on which these measures are defined. For example, mean and variance should be computed only when relying on interval scales.
In this talk, we will present our formal theory of IR evaluation measures, based on the representational theory of measurements, to determine whether and when IR measures are interval scales. We found that common set- based retrieval measures – namely Precision, Recall, and F-measure – always are interval scales in the case of binary relevance while this does not happen in the multi-graded relevance case. In the case of rank-based retrieval measures – namely AP, gRBP, DCG, and ERR – only gRBP is an interval scale when we choose a specific value of the parameter p and define a specific total order among systems while all the other IR measures are not interval scales. We will also introduce some brand new set-based and rank-based IR evaluation measures which ensure to be interval scales.
In our previous work we defined a theory of IR evaluation measures, based on the representational theory of measurement, which allowed us to determine whether and when IR measures are interval scales.
Finally, we will discuss the outcomes of an extensive evaluation, based on standard TREC collections, to study how our theoretical findings impact on the experimental ones. In particular, we conduct a correlation analysis to study the relationship among the above-mentioned state-of-the-art evaluation measures and their scales. We study how the scales of evaluation measures impact on non parametric and parametric statistical tests for multiple comparisons of IR system performance.


Nicola Ferro is associate professor in computer science at the University of Padua, Italy. His research interests include information retrieval, its experimental evaluation, multilingual information access and digital libraries. He is the coordinator of the CLEF evaluation initiative, which involves more than 200 research groups world-wide in large-scale IR evaluation activities. He was the coordinator of the EU Seventh Framework Programme Network of Excellence PROMISE on information retrieval evaluation. He is associate editor of ACM TOIS and was general chair of ECIR 2016.

NTCIR-14 Invited Talk 2

Date: June 11th (Tuesday), 2019 (Time: 17:00 p.m. - 18:00 p.m.)
Location: Hitotsubashi-hall, NII, Tokyo, Japan

Title: MediaEval 2019 Multimedia Benchmarking Initiative

Speaker: Gareth Jones (Dublin City University, Ireland)

Prof. Gareth Jones


MediaEval is a multimedia benchmarking initiative which seeks to evaluate new algorithms for multimedia access and retrieval. MediaEval emphasizes the "multi" in multimedia, including tasks combining various facet combinations of speech, audio, visual content, tags, users, and context. MediaEval innovates new tasks and techniques focusing on the human and social aspects of multimedia content in a community driven setting. The initiative provides a platform for researchers to organize benchmark tasks within a planned annual timeline and to report results at an end of campaign workshop. MediaEval 2019 marks the 10th anniversary of the foundation of MediaEval. This presentation will briefly overview 10 years of MediaEval campaigns and summarize current activities within the MediaEval 2019 campaign.


Gareth Jones conducts research on multiple topics in information retrieval, including multimedia, multilingual and personal content across a wide range of application areas. Over the last 20 years he has published hundreds of papers describing this work at multiple venues. Much of his research encompasses the design of tasks for the evaluation of this research, including test collections and evaluation metrics. Since 2002 he has been responsible for the organisation of international benchmarking tasks at venues including CLEF, FIRE, NTCIR and TRECVid. In 2010, together with Martha Larson, Delft University of Technology, The Netherlands, he co-founded the MediaEval Multimedia Benchmarking initiative to provide a platform for the development and evaluation of novel tasks in multimedia indexing and search. Gareth has served as co-Programme Chair for ECIR 2011, Information Retrieval Chair for ACM CIKM 2010, and co-Chair of ACM SIGIR 2013 and CLEF 2017 (with MediaEval 2017) both hosted in Dublin.
Gareth is a faculty member of the School of Computing, Dublin City University (DCU), Ireland and a Principal Researcher in the SFI ADAPT Centre. He holds B.Eng. and PhD degrees from the University of Bristol, UK. He has previously held posts at the University of Cambridge and University of Exeter, U.K., and in 1997 was a Toshiba Fellow at the Toshiba Corporation Research and Development Center in Kawasaki, Japan.

Last modified: 2019-06-04