NTCIR

In the field of Information Retrieval, researchers can quickly and easily compare ranking algorithms using what are called test collections. These test collections are often created by community efforts such as Text Retrieval Conference (TREC), and they involve sets of search queries, sets of documents and their corresponding relevance values. While these test collections have become an integral part of IR research, the process of data creation involves significant efforts of manual annotations, which often makes it very expensive and time-consuming. As an alternative, recent studies have proposed the use of large language models (LLMs) to completely replace human assessors. However, while LLMs seem to somewhat correlate with human judgments, they are not perfect, and a complete replacement with LLMs is argued to be not fully trustable . In this talk, I will present the history, trends, and problems of building test collections. Then, I will present an effective method to balance manual annotations with LLM annotations, which helps to make a trustable, yet a budget-friendly test collection.

Rikiya Takehi is a third-year undergraduate student in the Computer Science and Communications Engineering Department at Waseda University, supervised by Dr. Tetsuya Sakai. For one year (until Aug. 2024), he was working as a guest researcher at the NIST retrieval group with Dr. Ellen Voorhees and Dr. Ian Soboroff. His research has focused on the evaluation in IR, and currently, he is also developing a Product Recommendation Track at TREC. His past work on evaluation was presented at SIGIR-AP last year. His other research topics includes offline reinforcement learning (counterfactual learning) targeting RecSys problems, on which he works with Cyber Agent AI Lab and Hakuhodo Technologies, and he also collaborates closely with Yuta Saito of Cornell University.

As AI technology increasingly integrates into our daily lives, the need for systems that are not only functional but also deeply attuned to human users becomes paramount. This talk will explore human-centered AI, a design approach that emphasizes user-centric systems by leveraging insights from psychology and data collected through advanced sensing technologies. By understanding cognitive states and emotional responses, human-centered AI creates adaptive experiences that respond to users' needs, fostering engagement and reducing cognitive load.

Drawing on examples from gaming and AR/VR, I will discuss how real-time data from sensors like eye tracking and electrodermal activity (EDA) can inform AI systems that dynamically adjust to users' mental states. I’ll also cover ethical considerations, including data privacy and the challenges of designing systems that adapt without being intrusive. Ultimately, this talk aims to highlight the potential of human-centered AI to foster more empathetic, responsive interactions between humans and technology.

Emma is a 4th year PhD Candidate (Computer Science) at RMIT University and Intern at NII researching the intersection between cognitive neuroscience, user experience and video games. Her work specifically looks at methods for improving non-player character companions through adaptive systems and psychophysiological data.

She is passionate about the validation and evaluation of tools used to measure cognitive states during video gameplay and believes the foundation that this research sits upon can be improved through the use of objective data and neuroscientific techniques.

Her other research experience includes AR interaction design of a tool used to train new workers on a production line, digital empathy systems, and the exploration of individual biomarkers of performance of navigation tasks in VR.

Biases—both human and algorithmic—pose critical challenges to the fairness and effectiveness of search and recommendation systems. Our research investigates how these biases are interwoven into users’ search interactions and system outputs, impacting relevance judgments, search behavior and satisfaction, and credibility assessments. Drawing from recent studies, I will explore how cognitive biases, such as decoy effect, reference dependence, and expectation confirmation biases, shape users’ interactions with and evaluations on Web search engines and recommender systems. We also examine the threshold priming effects in large language models (LLMs), revealing how these systems inherit and amplify human-like biases in information retrieval tasks. Beyond diagnosis, I introduce bias mitigation strategies at varying levels, including biased result obfuscation, adaptive ranking, and LLM-enhanced data augmentation, aimed at fostering fairer retrieval and recommendation outcomes. The talk and reported projects offer new insights into bias-aware user modeling and evaluation, and highlight the urgency of building responsible bias-resilient search and recommendation environments in the era of AI.

Dr. Jiqun Liu is an Assistant Professor of Data Science and an Affiliated Assistant Professor of Psychology at the University of Oklahoma. His research focuses on information retrieval, recommender systems, and human-AI interaction, with a particular emphasis on cognitive biases, bias-aware user modeling, and scalable bias mitigation strategies. Dr. Liu’s work bridges behavioral science and IR, and investigates how boundedly rational users interact with search engines, recommender systems, and large language models in information tasks. His research on bias-aware user modeling and IR evaluation received grant support from National Science Foundation (NSF) and Microsoft, and has been published at premier venues, such as ACM SIGIR, CHIIR, JASIST, IP&M, EMNLP, and TheWebConf. His work has also been introduced in a research monograph entitled A Behavioral Economics Approach to Interactive Information Retrieval: Understanding and Supporting Boundedly Rational Users by Springer Nature and presented through invited talks and tutorials to both academic audiences and tech industry practitioners.