NTCIR

Title: TBA

Ian Soboroff (NIST, USA)

Using LLMs as Assistants for Building Large Test Collections

Rikiya Takehi (Waseda University)

Abstract:
While test collections are integral to IR research, creating them demands extensive manual annotation, making the process expensive and time-consuming. As an alternative, the use of large language models (LLMs) for relevance assessments has gained significant attention. However, LLM judgments are not perfect, and a complete replacement with LLMs is often argued to be unreliable. Are LLMs therefore useless? To what extent can we trust them? How do we decide when to trust and when not to? In this talk, I will present an effective method to balance manual annotations with LLM annotations, which helps to make test collections that are both reliable and affordable.

Biography:
Rikiya Takehi is a fourth-year undergraduate in the Computer Science and Engineering Department at Waseda University, advised by Prof. Tetsuya Sakai. From 2023–24, he worked with Dr. Ian Soboroff and Ellen Voorhees as a guest researcher in the NIST Retrieval Group, focusing on evaluation for search and text. He co-organises the Product Search & Recommendations Track at TREC and has led work published at SIGIR-AP, ICLR, and SIGIR. His broader interests include recommender systems, evaluation of rankings, and off-policy evaluation and learning.

We are delighted to invite submissions for the Eleventh International Workshop on Evaluating Information Access (EVIA 2025), which will be held in conjunction with NTCIR 18 in Tokyo, Japan, June 10-13, 2025. Information Access technologies play a crucial role as intermediaries between human information needs and digital information resources. The reliable evaluation of these technologies has been long recognized as central to the advancement of the field. As information retrieval technologies continue to pervade, retrieval methods diversify, and retrieval tools are enhanced, the importance of effective, efficient, and innovative evaluation grows as well.

Authors are encouraged to submit regular research or position papers of 2-10 pages in ACM format. Position papers should have a clearly stated focus or perspective, or weigh a balance of perspectives. Accepted papers will be allocated speaking slots appropriate to their content and length. Publication at EVIA is considered archival. Accepted papers are assigned a DOI and will appear on the conference website. Therefore, regular research and position papers should not have been previously published in an archival venue. Papers that previously appeared in non-archival workshops or on preprint repositories (e.g., arXiv) are not considered to be previously published. Simultaneous submission or publication in another venue is permitted if that venue is non-archival and accepts simultaneous submissions or submissions of previously published papers.

The review process is double-blind, and authors are required to take all reasonable steps to preserve the anonymity of their submission.

Authors may also request to present a poster on previously published work that may be of interest to the EVIA community. If you are requesting to present a poster on previously published work, it is sufficient to provide an abstract that includes a link to the previous publication. You do not need to upload a copy of the paper. Depending on time availability, people presenting posters may also be allocated a short speaking slot.

Papers should address one or more of the following topics, or that explore any other topics related to information access evaluation:

- Creation of test collections or other evaluation environments
- Evaluation using large language models
- Evaluation of retrieval augment generation (RAG) systems
- Evaluation using crowdsourcing, implicit feedback, living labs, or other inferential methods
- User studies and the evaluation of human-computer interaction in information retrieval
- Evaluation measures and statistical issues in retrieval evaluation
- Evaluation methods for multilingual, multimedia, or mobile information access
- Evaluation issues in enterprise retrieval systems
- Other novel information access tasks and their evaluation

May 2, 2025: Submission deadline

May 23, 2025: Notification of acceptance

May 30, 2025: Camera-ready paper deadline

June 10-13, 2025: EVIA Workshop and NTCIR

NTCIR and EVIA will be primarily in person events, but we will be able to arrange remote participation for cases in which travel to Tokyo is not possible. We look forward to an engaging exchange of research ideas, insights, and discoveries at EVIA 2025, contributing to the continued evolution of the information access landscape. Please join us for this intellectually enriching event!

Proceedings and reports from the first nine workshops are available below:

EVIA 2007 website, EVIA 2007 online proceedings
EVIA 2008 online proceedings
EVIA 2010 website, EVIA 2010 online proceedings
EVIA 2011 website, EVIA 2011 online proceedings
EVIA 2013 website, EVIA 2013 online proceedings
EVIA 2014 website, EVIA 2014 online proceedings
EVIA 2016 website, EVIA 2016 online proceedings
EVIA 2017 website, EVIA 2017 online proceedings
EVIA 2019 website, EVIA 2019 online proceedings
EVIA 2023 website, EVIA 2023 online proceedings