NTCIR

Session 1

Chair: Qingyao AI

Session 2

Chair: Doug Oard

Speaker: Ian Soboroff (NIST, USA)

Title: Evaluating Systems that Generate Content.

Abstract: The astounding emergence of ChatGPT and other AI systems that generate content, and their apparently incredible performance, are an inspiration to the research community. The performance of these LLMs is so impressive it is widely supposed that we can use them to measure their own effectiveness! We have had evaluation methods for generated content, including question answering, summarization, and translation, and in this talk I dust them off and present both a historical view and how we might approach those methods today. tl;dr, we have a lot of work to do.

Moderator: Doug Oard

Panelists: Akiko Aizawa (NII), Inho Kang (Naver), Yiqun Liu (Tsinghua University), Paul Thomas (Microsoft)

• Nuo Chen, Jiqun Liu, Tetsuya Sakai and Xiao-Ming Wu, Decoy Effect in Search Interaction: A Pilot Study
• Fernando Diaz, Best-Case Retrieval Evaluation: Improving the Sensitivity of Reciprocal Rank with Lexicographic Precision
• Tetsuya Sakai. Fairness-based Evaluation of Conversational Search: A Pilot Study
• Uyen Lai, Gurjit S. Randhawa and Paul Sheridan, Heaps' Law in GPT-Neo Large Language Model Emulated Corpora

We are delighted to invite submissions for the Tenth International Workshop on Evaluating Information Access (EVIA 2023), which will be held in conjunction with NTCIR 17 in Tokyo, Japan, on December 12-15, 2023. Information Access technologies play a crucial role as intermediaries between human information needs and digital information resources. The reliable evaluation of these technologies has been long recognized as central to the advancement of the field. As information retrieval technologies continue to pervade, retrieval methods diversify, and retrieval tools are enhanced, the importance of effective, efficient, and innovative evaluation grows as well.

Authors are encouraged to submit short research papers (2-5 pages) describing preliminary work or work in progress, full research papers (6-9 pages) describing completed work, or position papers (4-9 pages) that address one or more of the following topics, or that explore any other topics related to information access evaluation:

- Creation of test collections or other evaluation environments
- Evaluation using crowdsourcing, implicit feedback, living labs, or other inferential methods
- User studies and the evaluation of human-computer interaction in information retrieval
- Evaluation measures and statistical issues in retrieval evaluation
- Evaluation of generative large language models
- Evaluation methods for multilingual, multimedia, or mobile information access
- Evaluation issues in enterprise retrieval systems
- Other novel information access tasks and their evaluation

October 2, 2023

Submission deadline

October 22, 2023

Notification of acceptance

November 3, 2023

Camera-ready paper deadline

December 12-15, 2023

EVIA Workshop and NTCIR

NTCIR and EVIA will be primarily in person events, but we will be able to arrange remote participation for cases in which travel to Tokyo is not possible. We look forward to an engaging exchange of research ideas, insights, and discoveries at EVIA 2023, contributing to the continued evolution of the information access landscape. Please join us for this intellectually enriching event!

Proceedings and reports from the first nine workshops are available below:

EVIA 2007 online proceedings
EVIA 2008 online proceedings
EVIA 2010 online proceedings
EVIA 2011 online proceedings
EVIA 2013 online proceedings
EVIA 2014 online proceedings
EVIA 2016 online proceedings
EVIA 2017 online proceedings
EVIA 2019 online proceedings