NTCIR

Generative information retrieval is a promising neural retrieval paradigm that integrates all information in a corpus into a single, consolidated model. It formulates document retrieval as a document identifier (docid) generation task, allowing for end-to-end optimization toward a unified global retrieval objective. The generative information retrieval paradigm comes with a range of interesting evaluation questions. How can we gain insights in the inner workings of such end-to-end learnable pipelines? How do we know whether the retrieval model has "indexed" a corpus correctly? A generative retrieval model’s interconnected nature means that even small errors or changes in one component can lead to outsized impacts on overall performance, making the debugging process more complex and time-consuming. How do we establish some level of trustworthiness, in terms of reliability, resilience, and reproducibility? Typically formulated as a sequence-to-sequence learning problem, generative information retrieval lends naturally itself to combinations with a range of long-term optimization goals that go beyond short-term accuracy-based retrieval success. What are meaningful ways of probing and assessing generative information retrieval models that are being trained for long-term beyond-accuracy goals? In the talk I will present a range of evaluation challenges related generative information retrieval, mostly questions, with some early and partial answers.

Maarten de Rijke is a Distinguished University Professor of Artificial Intelligence and Information Retrieval at the University of Amsterdam. His research is focused on designing and evaluating trustworthy technology to connect people to information, particularly search engines, recommender systems, and conversational assistants. He is also co-founder and the scientific director of the Innovation Center for Artificial Intelligence (ICAI), a national collaboration between academic, industrial, governmental, and societal stakeholders aimed at talent development, research, and impact in AI.

Those of us who work on Information Retrieval (IR) share several common perspectives that shape our work. First, we assume that the searcher knows what they are looking for. Well, we know that’s not always true, but it’s been a useful assumption. Second, we assume that what we’re searching for has some digital representation, whether it was born digital, digitized, or simply described using digital metadata. The NTCIR-18 SUSHI task this year makes it clear that’s not always true, but that’s not the first such case. Third, we assume that if an IR system can find things, that its job is to provide them. But of course there are cases in which some searchers should not see some things. Fourth, we assume that the searcher can recognize what they are looking for; that’s why we often cast our task as creating a ranked list. But we know that’s not always true, as in (for example) spoken interaction, where ranked lists aren’t too helpful. Fifth, we expect that the searcher will be satisfied once they have found what they are looking for. But we know from our work on Cross-Language IR, for example, that there are cases in which we can find things that the searcher simply can’t make sense of. In this talk, I will argue that there is much to be learned from these kinds of cases where our common perspectives break down, and I will suggest that in our new era of generative large language models it can be useful to think broadly about these kinds of questions.

Douglas W. Oard is a Professor and the incoming Interim Dean at the University of Maryland College of Information. Over his three decades of research in information retrieval he has worked on cross-language IR, speech retrieval, and search among sensitive content, among other “off the beaten path” topics; essentially this is a talk that has been 30 years in the making. And as a Visiting Professor at NII since 2014, he has been pleased to contribute to the development of NTCIR as one of the world’s premiere evaluation venues for information access technologies.

The research field of building offline test collections was turned on its head 2 1/2 years ago when researchers started to show that LLMs had the potential to judge the relevance of documents for an offline testing data set. This discovery prompted a flurry of research both utilising LLMs in this new role as well as investigating the abilities and the limits of LLMs. This panel will both reflect on the existing research to use LLMs for relevance and to discuss other potential uses of this powerful modelling system for creating or augmenting other parts of offline testing.
The panel will consider the following three topics.
・Using LLMs for relevance
・Using LLMs to generate queries
・Using LLMs for other aspects of offline evaluation, simulation of user interactions, etc