Workshop Home
CALL FOR PARTICIPATION
The Third NTCIR Workshop (2001/2002)
Evaluation of Information Retrieval, Q&A, and Summarization
September 2001 - October 2002
Meeting: October 8-10, 2002, NII, Tokyo Japan
enquiries: ntc-admin

The 3rd NTCIR Workshop was OVER.

For the current NTCIR Workshop (the latest in the series of NTCIR Workshops), visit   !

An evaluation workshop of Asian language text retrieval, Q&A, and text summarization will be held from September 2001 to October, 2002. Participation is invited from anyone interested in retrieval of vairous kind of text and cross-lingual information retrieval of Asian languages from large-scale collections, and Q&A and text summarization of Japanese texts.

WORKSHOP OBJECTIVES

• To encourage research in information retrieval, Q&A, and text summarization by providing reusable test collections.
• To provide a forum for research groups interested in comparing results and exchanging ideas or opinions in an informal atmosphere.
• To improve the quality of the test collections based on the feedback from participants.

DESCRIPTION OF THE COLLECTION (DOCUMENT USED)

NTCIR-3 is over. For information on data see the NTCIR data page.

Documents will be provided for the participants those who returned required signed user agreement forms.

CLIR Task(Cross-Lingual IR) (Newspaper articles published in Asia)

- Chinese: CIRB010, United Daily News (1998-1999)

- Korean: Korea Economic Daily (1994)

- Japanese: Mainichi Newspaper (1998-1999)*

- English: Taiwan News and Chinatimes English News (1998-1999), Mainich Daily News (1998-1999)*

Patent Retrieval Task

- Japanese patents: 1998-1999 (about 17GB)

- Japio patent abstracts: 1995-1999

- Patent Abstracts of Japan (English translations for Japio patent abstracts): 1995-1999

- Patolis test collection (34 search topics for Japanese patents and relevance assessment)

- Newspaper articles (Japanese/English/Traditional Chinese)

Question Answering Task (qac)

Japanese newspaper articles (Mainichi Newspaper 1998-1999)*

Automatic Text Summarization Task (tsc2)

Japanese newspaper articles (Mainichi Newspaper 1998-1999)*

Web Retrieval Task (web)

Web documents collected from mainly .jp domain. Size: Large (ca. 100GB), Small (ca.10GB)
Available at the" Open-Lab" in the NII. Text processing such as indexing must be done on the machines in the Lab.

TASK (CHALLENGE) DESCRIPTION

Below is a brief summary of the tasks envisaged for the Workshop. A participant will conduct one or more of the tasks or subtasks below. Participation in only one subtask (for example Japanese monolingual IR (J-J)) is available:

CLIR Task (Cross-Lingual IR) APPLICATION
Documents and topics will be provided in four languages.

• Multilingual CLIR (MLIR): Search document collection more than one languages by one of four languages of topics.excepting Korean documents.
• Bilingual CLIR (BLIR): Search of any two different languages as language and documents, excepting search of English documents
• Single Lanugage IR (SLIR): Monolingual Search of Chinese, Korea, or Japanese.

Patent Retrieval TaskAPPLICATION

• Main Task
  • Cross-language Cross-DB retrieval: retrieve patents in response to J/E/C newspaper articles associated with technology and commercial products.
  • Monolingual Associative Retrieval: retrieve patents associated with an input Japanese patent
  For each search topic (i.e., either a newspaper article or patent), participants submit a list of retrieved patents and passages associated with the topic.

• Optional task: Any research reports are invited on patent processing using the above data, including, but not limited to: generating patent maps, paraphrasing claims, aligning claims and examples, summarization for patents, clustering patents.

Question Answering Task (qac)APPLICATION

qac website: http://www.nlp.cs.ritsumei.ac.jp/qac/

• Task 1: System extracts five answers from the documents in some order. 100 questions. System is required to return support information for each answer of the questions. We assume the support information as a paragraph, 100 letter passage or document which includes the answer.
• Task 2: System extracts only one answer from the documents. 100 questions. Support information is required.
• Task 3: evaluation of a series of questions. The related questions are given for the 30 of questions of Task 2.

Automatic Text Summarization Task (tsc2) APPLICATION

tsc website;http://lr-www.pi.titech.ac.jp/tsc/

• Task A (single document summarization): Given the texts to be summarized and summarization lengthes, the participants submit summaries for each text in plain text format.

• Task B (multi-document summarization): Given a set of texts, the participants produce summaries of it in plain text format. The information which was used to produce the document set, such as queries, as well as summarization lengthes are given to the participants.

Web Retrieval Task (web)APPLICATION

• [A. Survey Retrieval] Survey retrieval is similar to the traditional Ad-hoc retrieval for scientific documents or newspapers, where the system performs searching using newly provided topics for a static document set. Both recall and precision are evenly weighted for the evaluation. Two types of subtasks are provided: the retrieval using the topics in the almost same format of the past NTCIR workshops ('A1. Topic Retrieval') and the one using relevant documents given ('A2. Similarity Retrieval'). The page is the basic unit for evaluation, however, evidential passages can be used for complementary evaluation. Here the evidential passages means a part of each relevant document that gives the evidence of relevance judgment, and the submission of them is not mandatory.

• [B. Target Retrieval] Target retrieval is to try to evaluate the effectiveness of the retrieval in the case the user requires just one answer or a few (e.g. a fact-type retrieval, a reteieval of a site top page), where precision should be emphasized. The runs will be submitted as the ranked top 10 documents retrieved for each topic, being attached with evidential passages (not mandatory). Several evaluation measures will be applied.

• [C. Optional Tasks] The participants can freely subscribe proposals using the document set used in sub-task A and B, according to their own research interests. The results are presented as the paper/poster in the NTCIR-3 workshop meeting. If the proposal can involve several participants, it can be adopted as a sub-task and investigated in the details. 'C1. Search results classification' and 'C2. Speech-Driven Retrieval' are examples of the optinal tasks.

WORKSHOP SCHEDULE

TYPES OF PATICIPATION

• A. FULL: Submit results and describe the system. The correspondence between the group name and the group ID will be announced.

• B. ANONYMOUS: Submit results. The details of the system may not be reported. The correspondence between the group name and the group ID is not announced. This category is mainly for the participants from the companies who have troubles to report the details.

The list of the participating groups will be made public although the evaluation results will be announced using the group IDs only. Whichever of the types of participation, every participating group must submit (1) paper(s) for the workshop proceedings, (2) a system description form which describes your system, and (3) bibliographicreferences and a copy of all your papers when you will publish a paper using NTCIR test collections.

APPLICATIONS

Online application is available at:
http://research.nii.ac.jp/ntcir/ntcir-ws3/application-en.html

If you will have trouble with online registration, please download the registration form for each task, complete and return it via e-mail, fax, or postal mail to;

NTCIR Project Office
National Institute of Informatics (NII)
2-1-2 Hitotsubashi, Chiyoda-ku,Tokyo 101-8430, Japan
email: ntc-secretariat
fax: +81-3-4212-2751 phone: +81-3-4212-2750

ENQUIRIES

• Please send email to Noriko Kando, program chair, at kando, or to NTCIR Project administrators (ntc-admin).
• For the details of a specific task, please contact each task's chairs and organizers. See Contact.

NOTES

• For the first and second days of the Workshop meeting, participation will be retricted to the active participating groups that have submited results and selected people from organizing agencies. The third day will be an open forum of the researchers who are interested in the topics.
• The proceedings will be published online as well as printed-form.
• Dissemination of the research results using the NTCIR collections other than in the Workshop's Proceedings is welcome. However, the conditions of participation preclude specific advertising claims based on the results using the Collection or the Workshop.
• International participants are welcome. Announcements will be in English and Japanese.
• The official language for the proceedings papers and presentation at the Workshop meeting in October, 2002 is English.
• *DOCUMENT USAGE: The period of permitted use of Mainichi Newspapers and Mainichi Daily News are from 2001-09-01 to 2003-09-30. For active participants who sumbit the results and who affiliated at the organization outside Japan will be able to extend the period up to 2008-09-30. After the permitted period will be terminated, the participants will have to delete all the document data. Those who want to use the data after the period can purchase the data from Mainichi Newspaper Co., and obtain the permission for research purpose use from the company. The permitted period may vary depends on task.

ntc-admin
Last modified: 2002-10-25