The 6th NTCIR Workshop
Brief Task Description［Japanese］
The NTCIR is a series of evaluation workshop to enhance the research in information access technologies, including text retrieval, cross-lingual information access, question answering, etc, by providing infrastructure of evaluation and research including large-scale re-usable test collections, evaluation metrics and methodologies, and a forum of researchers who are interested in exchanging research ideas and evaluation methodologies. The emphasis has been placed on, but not limited to, Japanese and other Asian languages, and cross-lingual applications between Asian languages and English. The workshops are periodical events which are held once per about one and a half years (about 18 months).
The detailed task and collection description will be available in each task's web site.
The 6th NTCIR Workshop selected the following 4 areas of research as "Tasks" and 1 area as a "pilot workshop"; Other "pilot tasks" can be started any time during the process of NTCIR-6. For details, please visit each task's call for participation below.
NTCIR-4 CLIR used Traditional Chinese, Korean, Japanese, English documents published in East Asia in 1998-1999. NTCIR-5 CLIR add Chinese, Korean and Japanese (in 2000-2001) newspaper articles. NTCIR-6 Collection uses the same documents with NTCIR-5 except English. Simplified Chinese may be added later. Use four test collections of NTCIR-3 CLIR through -6 CLIR and examine the cross-collection variability.
CLQA uses Chinese, Japanese, English newspaper articles published in 2000-2001. Korean is under consideration.
NTCIR-6 PATENT uses the same test collection as NTCIR-5 PATENT and USPTO Patent. The test collections used in PATENT Retrieval tasks at NTCIR-3 through NTCIR-5 are available for training and will be used for NTCIR-6 task as well. For the test collections of NTCIR-3, and NTCIR-4 the separate user agreement forms for research purpose use are needed.
NTCIR-6 QAC uses 2000-2001 Japanese News articles from 2 different news sources. The test collections used in QAC at NTCIR-3 through -5, are provided as training collections.SUBMISSION RAW DATA and EVALUATION RESULTS: For every active participants who submitted results will receive all the submitted runs of the task and their evaluated results using the metrics set by the task as soon as those data will be available. The purpose of this is that we invited all the task participants to discuss, analyze or examine the evaluation metrics and evaluation results of the task. Evaluation is very critical issue for all of the researchers. So please examine how the evaluation is done and how the metrics behave, or whether there are any methods to overcome the limitation of current practice of the evaluation. With your cooperation, we would like to obtain fruitful examination of the evaluation results and metrics.