The 3rd NTCIR Workshop was OVER. |
For the current NTCIR Workshop (the latest in the series of NTCIR Workshops), visit ! |
An evaluation workshop of Asian language text retrieval, Q&A, and text summarization will be held from September 2001 to October, 2002. Participation is invited from anyone interested in retrieval of vairous kind of text and cross-lingual information retrieval of Asian languages from large-scale collections, and Q&A and text summarization of Japanese texts.
Documents will be provided for the participants those who returned required signed user agreement forms.
CLIR Task(Cross-Lingual IR) (Newspaper articles published in Asia)
Patent Retrieval Task
Question Answering Task (qac)
Japanese newspaper articles (Mainichi Newspaper 1998-1999)*
Automatic Text Summarization Task (tsc2)
Japanese newspaper articles (Mainichi Newspaper 1998-1999)*
Web Retrieval Task (web)
Web documents collected from mainly .jp domain. Size: Large (ca. 100GB),
Small (ca.10GB)
Available at the" Open-Lab" in the NII. Text processing such
as indexing must be done on the machines in the Lab.
Below is a brief summary of the tasks envisaged for the Workshop. A participant will conduct one or more of the tasks or subtasks below. Participation in only one subtask (for example Japanese monolingual IR (J-J)) is available:
CLIR Task (Cross-Lingual IR) APPLICATION
Documents and topics will be provided in four languages.
Patent Retrieval TaskAPPLICATION
Question Answering Task (qac)APPLICATION
qac website: http://www.nlp.cs.ritsumei.ac.jp/qac/
Automatic Text Summarization Task (tsc2) APPLICATION
tsc website;http://lr-www.pi.titech.ac.jp/tsc/
Web Retrieval Task (web)APPLICATION
[A. Survey Retrieval] Survey retrieval is similar to the traditional Ad-hoc retrieval for scientific documents or newspapers, where the system performs searching using newly provided topics for a static document set. Both recall and precision are evenly weighted for the evaluation. Two types of subtasks are provided: the retrieval using the topics in the almost same format of the past NTCIR workshops ('A1. Topic Retrieval') and the one using relevant documents given ('A2. Similarity Retrieval'). The page is the basic unit for evaluation, however, evidential passages can be used for complementary evaluation. Here the evidential passages means a part of each relevant document that gives the evidence of relevance judgment, and the submission of them is not mandatory.
[B. Target Retrieval] Target retrieval is to try to evaluate the effectiveness of the retrieval in the case the user requires just one answer or a few (e.g. a fact-type retrieval, a reteieval of a site top page), where precision should be emphasized. The runs will be submitted as the ranked top 10 documents retrieved for each topic, being attached with evidential passages (not mandatory). Several evaluation measures will be applied.
[C. Optional Tasks] The participants can freely subscribe proposals using the document set used in sub-task A and B, according to their own research interests. The results are presented as the paper/poster in the NTCIR-3 workshop meeting. If the proposal can involve several participants, it can be adopted as a sub-task and investigated in the details. 'C1. Search results classification' and 'C2. Speech-Driven Retrieval' are examples of the optinal tasks.
DATE SCHEDULE NOTE 2001-06-25 Task Plan Meeting: TSC2, QAC finished 2001-07-23 Task Plan Meeting: CLIR, Patent Search and Web Search finished 2001-07-24
-- 2001-07-25IPSJ SIGFI: Special Session for NTCIR (including discussion on NTCIR WS 3 tasks) finished 2001-08-01 Formal CFP for NTCIR WS 3 finished 2001-09-30 Application Due finished 2001-10-01 Document release (newspaper): delayed to 2001-10-15 finished 2001-10
-- 2002-01Dry Run and Round-Table Discussion (depends on each task) finished 2001-12 Open Lab start: Web Retrieval finished 2001-12
-- 2002-03Formal Run (depends on each task) finished 2002-02
-- 2002-07Round-Table Discussion (depends on each task) finished 2002-07-01 Evaluation Results Delivery finished 2002-08-20 Paper for Working Note Due finished 2002-10-08
-- 2002-10-10NTCIR Workshop 3 Meeting
Days 1-2: Closed session
Day 3: Open sessionfinished 2002-12-01 Paper for Final Proceedings Due
A. FULL: Submit results and describe the system. The correspondence between the group name and the group ID will be announced.
B. ANONYMOUS: Submit results. The details of the system may not be reported. The correspondence between the group name and the group ID is not announced. This category is mainly for the participants from the companies who have troubles to report the details.
The list of the participating groups will be made public although the evaluation results will be announced using the group IDs only. Whichever of the types of participation, every participating group must submit (1) paper(s) for the workshop proceedings, (2) a system description form which describes your system, and (3) bibliographicreferences and a copy of all your papers when you will publish a paper using NTCIR test collections.
Online application is available at:
http://research.nii.ac.jp/ntcir/ntcir-ws3/application-en.html
If you will have trouble with online registration, please download the
registration form for each task, complete and return it via e-mail, fax,
or postal mail to;
NTCIR Project Office
National Institute of Informatics (NII)
2-1-2 Hitotsubashi, Chiyoda-ku,Tokyo 101-8430, Japan
email: ntc-secretariat
fax: +81-3-4212-2751 phone: +81-3-4212-2750