[Date Prev][Date Next][Date Index]
[ntcir:194] CFP for NTCIR-6
- To: ntcir@xxxxxxxxx
- Subject: [ntcir:194] CFP for NTCIR-6
- From: Noriko Kando <kando@xxxxxxxxx>
- Date: Tue, 18 Apr 2006 17:08:04 +0900
Call for Participation to NTCIR-6
Evaluation of Information Access Technologies: IR, QA and
Cross-lingual Information Access Technologies
Final Meeting: 15-18 May 2007, NII, Tokyo, Japan
The NTCIR is a series of evaluation workshop to enhance the
research in information access technologies, including text
retrieval, question answering, cross-lingual information
access, etc, by providing infrastructure of evaluation
and research including test collections, evaluation metrics
and methodologies, and a forum of researchers.
NTCIR has used East Asian Language documents but attracted
international participation. For example the previous NTCIR
has participants from 15 different countries and areas including
Australia, Canada, China PRC, Germany, Hong Kong, Ireland, Japan,
Korea, Netherlands, Singapore, Spain, Switzerland, Taiwan ROC,
UK, United States.
NTCIR-6 selected the following 4 areas of research as
"Tasks" and 1 area as a "pilot workshop"; Other "pilot tasks"
can be started any time during the process of NTCIR-6.
1. Cross-Lingual Information Retrieval Task (CLIR)
Multi- and Bi-lingual CLIR, and
Single language IR
Languages: Traditional Chinese, Korean, and Japanese.
Simplified Chinese can be added. To conclude the CLIR to
news documents, 4 CLIR test collections, NTCIR-3 through -6
will be used and Cross-collection analysis will be done.
New metrics for graded relevance judgments are used.
Discussion about evaluation metrics and methodologies
are also welcome.
2. Cross-Language Question Answering Task (CLQA)
Focus on Named Entities, which are one of the problems
in CL information access in Asian context.
Traditional Chinese, English and Japanese. Korean language
is under consideration. Volunteer for cooperation to
organize Korean part is welcome.
3. Patent Retrieval Task (PATENT)
Retrieval task: "Invalidity search" Using Japanese patents
(10 year patent application fulltexts) and US patents.
Use 4 patent collections of NTCIR-3 through -6.
Classification task: Multi-viewpoint categorization. The
purpose is to categorize target patent applications based
on the F-term classification system.
4. Question Answering Task (QAC)
Question answering beyond factoid questions and their evaluation
method. Focusing to complicated questions like "WHY".
5. Pilot Tasks
Any attempts to test the problems to be solve in short
time, or feasibility studies for the future tasks. To be
announced later. Opinion Extraction, Multilingual Multi-
document Summarization, Evaluation of WEB Search engine
were now under consideration, but not decided
6. Pilot Workshop: Multimodal Summarization of Trend Information (MuST)
Focusing numerical information in the text. Extract numerical
information representing the trends from multiple documents,
analyze, summarize and display visullay. We constructed an annotated
corpus and share it. Research groups from various disciplinaries share
the corpus and work various direction of research using the corpus.
Currently using Japanese documents only. To be announced later.
WHAT'S NEW to NTCIR-6:
(1) Cross-Collection Evaluation:
In IR related tasks, multiple test collections for the same
document genres and the same users' tasks will be used and
analyze the variability across the collections.
(2) Submission Raw Data and Evaluation Results:
All the submitted runs and their evaluated results using the metrics
set by the task will be available for the active participants of the
task. The purpose of this is that we invited all the task participants
to discuss, analyze or examine the evaluation metrics and evaluation
results of the task.
Evaluation is very critical issue for all of the researchers. So
please examine how the evaluation is done and how the metrics behave,
or whether there are any methods to overcome the limitation of
current practice of the evaluation. With your cooperation, we would
like to obtain fruitful examination of the evaluation results and metrics.
OPEN SUBMISSION SESSION:
Conjuction with NTCIR-6 Final Meeting, the separate conference called
"Open Submission Session" will be held. This is a refereed international
conference on research in Information Access and their evaluation.
Detailed information will be announced later.
Registration Deadline: 2006-05-31
Document Set Release: 2006-06-01
Dry Run: from 2006-07 to 2006-09
Formal Run： from 2006-10 to 2006-12
Evaluation Results Return: by 2007-02-01
Paper for the Proceedings Due : 2007-03-01
Final Meeting, Tokyo, Japan: 2007-05-15/18
** ONLINE REGISTRATION IS AVAILABLE NOW !!
Noriko Kando at kando (at) nii.ac.jp
Your participation is more than welcome. Please join and work together!