NTCIR

Many data science applications that use machine learning techniques depend on humans providing the initial data set so algorithms can process the rest or to evaluate the performance of such algorithms. Not only can labeled data for training and evaluation be collected faster, cheaper, and easier than ever before, but we now see the emergence of novel infrastructure that combines computations performed by humans and machines. Building these labeling pipelines remain difficult and these difficulties need to be addressed by practitioners and researchers to advance the state of the art. In this talk, I’ll outline things that work in practice and describe a number of trade-offs when designing and implementing computation systems that use humans and machines.

Omar is a Principal Data Scientist Lead at Microsoft in Silicon Valley where he works on the intersection of social networks, temporal information, knowledge graphs, and human computation. He has shipped many features for Bing and other Microsoft properties. He is the co-chair for the new IR system-oriented conference, called DESIRES.

In this talk I will describe a research program called MAchine Translation for English Retrieval of Information in Any Language (MATERIAL) that includes a substantial focus on Cross Language Information Retrieval (CLIR). Over four years, this program expects to build new CLIR test collections for ten new languages, in each case with English queries. Novel aspects of these test collections will include (1) domain-limited, sense-specific, and morphology-specific queries, and (2) mixed collections including both text and speech. Two novel aspects of the evaluation design are a focus on set-based rather than ranked retrieval, and the use of a linear utility measure for evaluating result set selection. The MATERIAL program also includes an interactive CLIR evaluation in which assessors use system-generated English summaries in an effort to identify the truly relevant documents in the result set. In this talk I will start by walking through these evaluation design issues, and then I will offer our initial thoughts on the consequences of these evaluation choices for our system designs. Additional information on the MATERIAL program is available at https://www.iarpa.gov/index.php/research-programs/material.

Douglas Oard is a Professor at the University of Maryland, College Park (USA), with joint appointments there in the College of Information Studies (Maryland’s iSchool) and the University of Maryland Institute for Advanced Computer Studies (UMIACS). He is also a visiting professor at the National Institute of Informatics (Japan). Dr. Oard’s research interests center around the use of emerging technologies to support information seeking by end users. Additional information is available at http://terpconnect.umd.edu/~oard/.

My participation in the first NTCIR workshop starting in 1997 led to a fifteen year research journey. It was a journey through Kanji, Katakana, Hiragana, segmentation, bigrams, phonetic recognition, decompounding and parallel corpora alignment for lexicon development. In languages the travels went through Japanese, Chinese, Korean, European languages, Russian, Arabic, and Hindi, as well as mathematics. It produced three SIGIR workshops (2002, 2006, 2009) on cross-language search and multilingual information access as well as a special issue of Information Processing and Management on cross language information retrieval. In later years the research direction changed toward geographic and geo-temporal information retrieval evaluation. This talk will cover highlights of my personal research journey and pay tribute to colleagues and students with whom I have been fortunate enough to collaborate.

After receiving a Master's degree in Mathematics from UC Berkeley n 1964, Fredric Gey worked for 3 1/2 years at Bell Laboratories (later ATT Labs Research). In 1967 he returned to Berkeley and worked for 21 years as a staff scientist at Lawrence Berkeley Laboratory in the Computer Science Research Department. In 1989 he returned to the Berkeley campus as Data Archivist and Librarian for Social Science and Health Statistics for UC Berkeley, while simultaneously pursuing a PhD in Information Science, which was conferred in 1993. His dissertation "Probabilistic Dependence and Logistic Inference in Information Retrieval" developed the first of several logistic regression ranking models which have stood the test of time. In 1996 he received a USA National Science Foundation Grant to develop logistic regression search models. In 1998 he turned the research direction of this grant toward cross-language information retrieval, leading to participation in the first NTCIR workshop in 1997-1999, and in 2000 to CLEF, the Cross Language Evaluation Forum for European languages. In addition to multilingual information access, he has also done research and development in nuclear forensics, geographic information retrieval, digital humanities and social science information systems. He was the General Chair of ACM-SIGIR 1999, the 22nd International Conference on Research and Development in Information Retrieval.

In the last part of the twentieth century, the idea of shared task evaluation emerged as a significant force shaping information retrieval research. In this talk, I will start by tracing the evolution of that idea at it moved around the world from its inception at Cambridge through its incubation at the Text Retrieval Conference, to its present incarnation at NTCIR. I’ll then look back over the history of NTCIR to highlight some of the impactful and innovative evaluation tasks that have been invented there, exploring the impact of each from national, regional, or global perspectives. I’ll wrap up with a few remarks on how the role of shared task evaluation in information retrieval research is evolving today, and what that might suggest for the future impact of NTCIR.

Douglas Oard is a Professor at the University of Maryland, College Park (USA), with joint appointments there in the College of Information Studies (Maryland’s iSchool) and the University of Maryland Institute for Advanced Computer Studies (UMIACS). He is also a visiting professor at the National Institute of Informatics (Japan) and a former general chair of NTCIR. Dr. Oard’s research interests center around the use of emerging technologies to support information seeking by end users. Additional information is available at http://terpconnect.umd.edu/~oard/.