NTCIR Home > NTCIR-11 > Evaluation Tasks
The Eleventh NTCIR Workshop (NTCIR-11) Task Selection Committee has selected the following six Core Tasks and three Pilot Tasks.
For details and latest information, please visit each task’s homepage.
For contact to each task, please see here, too.
IMine MATH MedNLP MobileClick RITE-VAL SpokenQuery&Doc QA Lab Temporalia RecipeSearch
CORE TASKS
Mining users’ underlying intents of a query is an interesting topic
for both IR communities and commercial search engines. The NTCIR-11 IMine
task aims to provide common data sets and evaluation methodology to
researchers for exploring the technologies of mining and satisfying
different user intents behind a Web search query. IMine is short for search
Intent Mining and it also pronounces like "曖昧" which means "ambiguous" in
Japanese. IMine is a core task of NTCIR-11 and also a succeeding work of
INTENT@NTCIR-9 and INTENT@NTCIR-10 tasks.
Task Website: http://www.thuir.org/IMine/
The NTCIR Math Task aims to develop an evaluation test collection for mathematical formulae search, in order to facilitate and encourage research in mathematical information retrieval and its related fields. In the NTCIR-11 Math-2 Task, we will continue to pursue our initial goal to create an active and emerging community in Math IR. Using the same document collection, we plan to develop a new query set that is suitable
both for math/text retrieval evaluations. For updated information,
please refer to http://ntcir-math.nii.ac.jp/.
Recently, medical records are increasingly written on electronic media instead of on paper, thereby increasing the importance of information processing in medical fields. In this proposed core challenge task, participants are supposed to retrieve important information from medical documents in Japanese and normalize them, which are the elemental technologies to develop computational systems for supporting a wide range of medical services.
Task Website: http://mednlp.jp/ntcir11/
Current web search engines usually return a ranked list of URLs in
response to a query. The user often has to visit several web pages and
locate relevant parts within long web pages. Especially for mobile
users, those actions need much effort and attention. However, for some
classes of queries, the system should be able to return a very concise
summary of relevant information directly and to satisfy her immediately
after clicking on the search button. In the MobileClick task, systems
are expected to output a concise summary of information relevant to a
given query and to provide immediate and direct information access for
mobile users. For more information, please visit:
http://www.dl.kuis.kyoto-u.ac.jp/ntcir-11/mobileclick/.
NTCIR-11 RITE-VAL is a generic benchmark task that addresses common semantic processing needs in various NLP/Information Access research areas. We will evaluate systems which automatically detect entailment, paraphrase, and contradiction in texts written in Japanese, Simplified Chinese, Traditional Chinese, and English. RITE-VAL following RITE-1 and RITE-2 will present the main two subtasks: Fact Validation (Search) and System Validation (Unit test). In Search subtask, your system identifies whether a given text is entailed from the sentences relevant to it, which are retrieved from Wikipedia or textbooks. Unit test subtask provides a data set that includes a breakdown of linguistic phenomena that are necessary for recognizing relations between two texts. Since RITE is a friendly evaluation effort, we welcome your participation even with an ambitious/exploratory approach.
Task Website: https://sites.google.com/site/ntcir11riteval/
Current information retrieval framework seems to face bottleneck in its
human interface for drawing out one's information need. SpokenQuery&Doc
task tries to overcome it by making use of spontaneously spoken queries.
One of the advantage of the use of speech as an input method to retrieval
systems is that it enables users to easily submit long queries to give
systems rich clues for retrieval, because the unconstrained speech is common
in daily use for human and the most natural and easy method to express
one's thought. The target document collection is also spoken documents.
Since the spoken document retrieval has been evaluated in the previous
SpokenDoc-1 and 2, SpokenQuery&Doc can be considered as an extension
of the previous SpokenDoc tasks, by introducing spoken queries instead
of text queries.
For more information, please visit: http://www.nlp.cs.tut.ac.jp/ntcir11/
PILOT TASK
Task Website: http://ntcir.nii.ac.jp/QALab/
Given the fact that time plays crucial role in estimating infromation relevance and validity, we believe that sussessful search engines must consider temporal aspects of information in greater detail. We set two subtasks for this purpose: Temporal Query Intent Classification (TQIC) and Temporal Information Retrieval (TIR). TQIC asks participants to classify a given query to several temporal classes. TIR asks participants to retrieve a set of documents that are relevant to information needs that incorporate time factor.
Task Website: https://sites.google.com/site/ntcirtemporalia/
Task Website (in NTCIR CMS Site): http://ntcir.nii.ac.jp/Temporalia/NTCIR-11-Temporalia/
Mailing List: https://groups.google.com/forum/#!forum/ntcirtemporalia
Facebook Page: https://www.facebook.com/ntcirtemporalia
Twitter: https://twitter.com/ntcirtemporalia
Information access tasks involving food have traditionally focused on locating or ranking relevant restaurants given a user’s information need.
However, home cooking remains a fundamental method for acquiring a meal. In this pilot, we propose exploring the information access tasks associated with food recipes.
Japanese/English recipe search collections will be constructed.
Task Website: https://sites.google.com/site/ntcir11recipesearch/
Contact to Task Organizers: ntcir11.recipesearch
Last Modified: 2014-07-09
PAGE TOP