NTCIR

In the field of Information Retrieval, researchers can quickly and easily compare ranking algorithms using what are called test collections. These test collections are often created by community efforts such as Text Retrieval Conference (TREC), and they involve sets of search queries, sets of documents and their corresponding relevance values. While these test collections have become an integral part of IR research, the process of data creation involves significant efforts of manual annotations, which often makes it very expensive and time-consuming. As an alternative, recent studies have proposed the use of large language models (LLMs) to completely replace human assessors. However, while LLMs seem to somewhat correlate with human judgments, they are not perfect, and a complete replacement with LLMs is argued to be not fully trustable . In this talk, I will present the history, trends, and problems of building test collections. Then, I will present an effective method to balance manual annotations with LLM annotations, which helps to make a trustable, yet a budget-friendly test collection.

Rikiya Takehi is a third-year undergraduate student in the Computer Science and Communications Engineering Department at Waseda University, supervised by Dr. Tetsuya Sakai. For one year (until Aug. 2024), he was working as a guest researcher at the NIST retrieval group with Dr. Ellen Voorhees and Dr. Ian Soboroff. His research has focused on the evaluation in IR, and currently, he is also developing a Product Recommendation Track at TREC. His past work on evaluation was presented at SIGIR-AP last year. His other research topics includes offline reinforcement learning (counterfactual learning) targeting RecSys problems, on which he works with Cyber Agent AI Lab and Hakuhodo Technologies, and he also collaborates closely with Yuta Saito of Cornell University.