NTCIR

To state the obvious, 2020 was an unusual year. The pandemic that caused so many changes large and small impacted the Text REtrieval Conference (TREC), too: TREC implemented its first extra-curricular track (TREC-COVID); both relevance assessing and the conference itself were required to be remote; and many existing TREC tracks either pivoted to be completely focused on COVID-19 or included some topics regarding it. This talk will describe implementing TREC in 2020 and highlight the outcomes of its eight tracks.

Ellen Voorhees is a Senior Research Scientist at the US National Institute of Standards and Technology (NIST). Her primary responsibility at NIST is to manage the Text REtrieval Conference (TREC) project, a project that develops the infrastructure required for large-scale evaluation of search engines and other information access technology. Voorhees' research focuses on developing and validating appropriate evaluation schemes to measure system effectiveness for diverse user tasks.

Voorhees received a B.Sc. in computer science from the Pennsylvania State University, and M.Sc. and Ph.D. degrees in computer science from Cornell University. Prior to joining NIST she was a Senior Member of Technical Staff at Siemens Corporate Research in Princeton, NJ where her work on intelligent agents applied to information access resulted in three patents. She is a fellow of the ACM, a member of AAAI, and has been elected as a fellow of the Washington Academy of Sciences. She has published numerous articles on information retrieval techniques and evaluation methodologies and serves on the review boards of several journals and conferences.

The initial part of this talk will discuss the achievements and happenings in CLEF, the European initiative whose main mission is to promote research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure. We will focus on the just concluded CLEF 2020 edition
( https://clef2020.clef-initiative.eu/) and the just started CLEF 2021 edition
( http://clef2021.clef-initiative.eu/).

The second part of the talk will present the Covid-10 MLIA @ Eval initiative ( http://eval.covid19-mlia.eu/), a voluntary effort supported and promoted by several communities, among which CLEF. Covid-19 MLIA @ Eval is a community effort to boost the development of (language) resources and Multilingual Information Access (MLIA) systems specifically tailored on Covid-19. In particular, we organise evaluation tasks that steer the development of systems and resources in the following areas: information extraction, multilingual semantic search, and machine translation.

Nicola Ferro ( http://www.dei.unipd.it/~ferro/ ) is full professor of computer science at the University of Padua, Italy. His research interests include information retrieval, its experimental evaluation, multilingual information access and digital libraries and he published more than 350 papers on these topics. He is co-organizer of the Covid-19 MLIA @ Eval initiative and he is the chair of the CLEF evaluation initiative, which involves more than 200 research groups world-wide in large-scale IR evaluation activities. He was the coordinator of the EU 7FP Network of Excellence PROMISE on information retrieval evaluation. He is associate editor of ACM TOIS and was general chair of ECIR 2016, and short papers program co-chair of ECIR 2020.

MediaEval is a multimedia benchmarking initiative which seeks to evaluate new algorithms for multimedia access and retrieval. MediaEval emphasizes the "multi" in multimedia, including tasks combining various facet combinations of speech, audio, visual content, tags, users, and context. MediaEval innovates new tasks and techniques focusing on the human and social aspects of multimedia content in a community driven setting. The initiative provides a platform for researchers to organize benchmark tasks within a planned annual timeline and to report results at an end of campaign workshop. This presentation will overview the objectives of the MediaEval campaigns and summarize current activities within MediaEval 2020.

Gareth Jones conducts research on multiple topics in information retrieval, including multimedia, multilingual and personal content across a wide range of application areas. Over the last 20 years he has published hundreds of papers describing this work at multiple venues. Much of his research encompasses the design of tasks for the evaluation of this research, including test collections and evaluation metrics. Since 2002 he has been responsible for the organisation of international benchmarking tasks at venues including CLEF, FIRE, NTCIR and TRECVid. In 2010, together with Martha Larson, Radboud University, The Netherlands, he co-founded the MediaEval Multimedia Benchmarking initiative to provide a platform for the development and evaluation of novel tasks in multimedia indexing and search. Gareth has served as co-Programme Chair for ECIR 2011, Information Retrieval Chair for ACM CIKM 2010, and co-Chair of ACM SIGIR 2013 and CLEF 2017 (with MediaEval 2017) both hosted in Dublin.

Gareth is a faculty member of the School of Computing, Dublin City University (DCU), Ireland and a Principal Researcher in the SFI ADAPT Centre. He holds B.Eng. and PhD degrees from the University of Bristol, UK. He has previously held posts at the University of Cambridge and University of Exeter, U.K., and in 1997 was a Toshiba Fellow at the Toshiba Corporation Research and Development Center in Kawasaki, Japan.

Technology advances mean that we can now gather detailed multimedia traces that model our life activities; these are called lifelogs. As research began in this domain over fifteen years ago, it was noticeable that there was little understanding of how lifelogs can positively impact on the individual/society nor did our research community understand how lifelogs can be organised and indexed to provide effective retrieval facilities. This talk will provide an overview of the outputs of the NTCIR-Lifelog task. The motivation for proposing the NTCIR-Lifelog task and the progress made by running this task three times at NTCIR will be discussed along with highlighting the wider impact of NTCIR-Lifelog in terms of related benchmarking activities. Finally, future plans for next generation lifelog-related tasks at NTCIR will be proposed along with a possible roadmap for the future years.

Cathal Gurrin is an associate professor and deputy department head at the School of Computing, at Dublin City University (DCU), Ireland and he is an investigator at the Insight Centre for Data Analytics and the Adapt Centre, both at DCU. His research interests are personal analytics and lifelogging, which integrate personal sensing, computer science, cognitive science and data-driven healthcare analytics to realise the next-generation of digital records for the individual. He regularly speaks at Quantified Self events and his research been featured internationally on Discovery Channel, BBC, NHK, as well as in the Economist magazine, New York Times, among many others. He has been the General Chair of ECIR 2011, MMM 2014, MB2016 and MMM2017, CBMI2019 & ICMR2020. He is the author of Lifelogging: Personal Big Data from the FNTIR series.