Brief Task Description

The NTCIR is a series of evaluation workshop to enhance the research in information access technologies, including text retrieval, cross-lingual information access, question answering, etc, by providing infrastructure of evaluation and research including large-scale re-usable test collections, evaluation metrics and methodologies, and a forum of researchers who are interested in exchanging research ideas and evaluation methodologies. The emphasis has been placed on, but not limited to, Japanese and other Asian languages, and cross-lingual applications between Asian languages and English. The workshops are periodical events which are held once per about one and a half years (about 18 months).

The detailed task and collection description will be available in each task's web site.

TASKS:

The 6th NTCIR Workshop selected the following 4 areas of research as "Tasks" and 1 area as a "pilot workshop"; Other "pilot tasks" can be started any time during the process of NTCIR-6. For details, please visit each task's call for participation below.

1. Cross-Lingual Information Retrieval Task (CLIR)

Multilingual CLIR

Bilingual CLIR, and

Single language IR

Languages: Traditional Chinese, Korean, and Japanese. Simplified Chinese can be added. To conclude the CLIR to news documents, 4 test collections, NTCIR-3 through -6 for news documents will be used and Cross-collection analysis will be done. New metrics for graded relevance judgments are used. Discussion about evaluation metrics and methodologies are also welcome.

2. Cross-Language Question Answering Task (CLQA)

Focus on Named Entities, which are one of the problems in CL information access in Asian context.

5 subtasks (C->C, E->C, C->E, E->J, J->E). (J->J is covered in QAC). Korean language is under consideration. Comments and volunteer for cooperation to organize Korean part is welcome.

3. Patent Retrieval Task (PATENT)

Retrieval task: "Invalidity search" Using Japanese patents and US patents

Classification task: Multi-viewpoint categorization. The purpose is to categorize target patent applications based on the F-term classification system.

4. Question Answering Task (QAC)

Question answering beyond factoid question and their evaluation method. Focusing to complicated questions like "WHY". Currently plan to use Japanese documents only.

5. Pilot Tasks

Any attempts to test the problems to be solve in short time, or feasibility studies for the future tasks. To be announced. Opinion Extraction, Multilingual Multi-document Summarization, Evaluation of WEB Search engine are now under consideration.

6. Pilot Workshop

MuST: Multimodal Summarization of Trend Information -- Currently using Japanese documents only

TEST COLLECTIONS:

NTCIR-4 CLIR used Traditional Chinese, Korean, Japanese, English documents published in East Asia in 1998-1999. NTCIR-5 CLIR add Chinese, Korean and Japanese (in 2000-2001) newspaper articles. NTCIR-6 Collection uses the same documents with NTCIR-5 except English. Simplified Chinese may be added later. Use four test collections of NTCIR-3 CLIR through -6 CLIR and examine the cross-collection variability.

CLQA uses Chinese, Japanese, English newspaper articles published in 2000-2001. Korean is under consideration.

NTCIR-6 PATENT uses the same test collection as NTCIR-5 PATENT and USPTO Patent. The test collections used in PATENT Retrieval tasks at NTCIR-3 through NTCIR-5 are available for training and will be used for NTCIR-6 task as well. For the test collections of NTCIR-3, and NTCIR-4 the separate user agreement forms for research purpose use are needed.

NTCIR-6 QAC uses 2000-2001 Japanese News articles from 2 different news sources. The test collections used in QAC at NTCIR-3 through -5, are provided as training collections.

SUBMISSION RAW DATA and EVALUATION RESULTS: For every active participants who submitted results will receive all the submitted runs of the task and their evaluated results using the metrics set by the task as soon as those data will be available. The purpose of this is that we invited all the task participants to discuss, analyze or examine the evaluation metrics and evaluation results of the task. Evaluation is very critical issue for all of the researchers. So please examine how the evaluation is done and how the metrics behave, or whether there are any methods to overcome the limitation of current practice of the evaluation. With your cooperation, we would like to obtain fruitful examination of the evaluation results and metrics.

HOW TO PARTICIPATE:

Please consult How to Participate. Online registration form is linked from the "How to Participate". After registration, submitting the signed user agreement forms is needed.

---