The NTCIR Workshop is a series of evaluation workshops designed to enhance research in Information Access (IA) technologies including information retrieval, question answering, text summarization, extraction, etc. It was co-sponcered by Japan Society for Promotion of Science (JSPS) as part of JSPS "Research for Future" Program" and National Center for Science Information Systems (NACSIS) since 1997, by JSPS and Research Center for Information Resources at National Institute of Informatics (RCIR/NII,) in FY 2000, and by MEXT Grant-in-Aid for Scientific Research on Priority Areas of "Informatics" (#13224087) and RCIR/NII in and after FY2001.
The aims are;
An evaluation workshop usually provides test collections (data sets usable for experiments) and unified evaluation procedures for experiment results. Each participating group conducts research and experiments using the common data provided by the NTCIR organizer with various approaches. The importance of reusable large-scale standard test collections in IA research has been widely recognized and an evaluation workshop is now recognized as a new style of active research project that facilitates research by providing the data and a forum for research idea exchange and technology transfer.
For the First NTCIR Workshop, the process was started from November, 1998, and the Workshop meeting was held on August 30 - September 1, 1999, at KKR Hotel, Tokyo. Twenty-eight groups from six countries conducted the tasks and submitted the results for the first workshop. For the Second Workshop, the process was started from June 2000 and the meeting was held on March 7-9, 2001, NII, Tokyo and forty-six groups from eight countries have registered for the tasks and 36 groups conducted and submitted the results to one or more tasks. The process of the Third NTCIR Workshop started from October 2001 and the meeting was held on October 8-10, 2002, NII, Tokyo and sixty-five groups from nine countries submitted the results.
From the beginning of the NTCIR Project, We have looked at both traditional laboratory-typed IR system testing and evaluation of more challenging technologies. For the laboratory-typed testing, we have placed emphasis on (1) information retrieval (IR) with Japanese or other Asian languages and (2) cross-lingual information retrieval. For the challenging issues, (3) shift from document retrieval to "information" retrieval and technologies to utilizing information in the documents, and (4) investigation for realistic evaluation, including evaluation methods for summarization, multigrade relevance judgments and single-numbered averageable measures for such judgments, evaluation methods suitable for retrieval and processing of particular document-genre and its usage of the user group of the genre and so on.
The test collection constructed, tasks, participants, sponcership for the previous NTCIR workshops are as follows;
Table 1. Tasks, Collections and Participants of the Previous Workshops
Table 2.The test collections constructed (and made available) or will be constructed through NTCIR Workshops
JA: Japanese, EN: English, CH: Chinese (tr: traditional, sm: simplified), KO: Korean
* Documents are avaible for research purpose use from Nichigai Associates, Co. (for Japanese users) or MAINICHI International, Inc. (for international users).
The details of the NTCIR-4 Test Collections are available HERE