Evaluation Tasks

The Eleventh NTCIR Workshop (NTCIR-11) Task Selection Committee has selected the following six Core Tasks and three Pilot Tasks.
For details and latest information, please visit each task’s homepage.
For contact to each task, please see here, too.

IMine    MATH    MedNLP    MobileClick    RITE-VAL    SpokenQuery&Doc    QA Lab    Temporalia    NEWRecipeSearch


Search Intent and Task Mining ("IMine")

Mining users’ underlying intents of a query is an interesting topic for both IR communities and commercial search engines. The NTCIR-11 IMine task aims to provide common data sets and evaluation methodology to researchers for exploring the technologies of mining and satisfying different user intents behind a Web search query. IMine is short for search Intent Mining and it also pronounces like "曖昧" which means "ambiguous" in Japanese. IMine is a core task of NTCIR-11 and also a succeeding work of INTENT@NTCIR-9 and INTENT@NTCIR-10 tasks.
Task Website: http://www.thuir.org/IMine/

Mathematical Information Access ("Math-2")

The NTCIR Math Task aims to develop an evaluation test collection for mathematical formulae search, in order to facilitate and encourage research in mathematical information retrieval and its related fields. In the NTCIR-11 Math-2 Task, we will continue to pursue our initial goal to create an active and emerging community in Math IR. Using the same document collection, we plan to develop a new query set that is suitable both for math/text retrieval evaluations. For updated information,
please refer to http://ntcir-math.nii.ac.jp/.

Medical Natural Language Processing ("MedNLP-2")

Recently, medical records are increasingly written on electronic media instead of on paper, thereby increasing the importance of information processing in medical fields. In this proposed core challenge task, participants are supposed to retrieve important information from medical documents in Japanese and normalize them, which are the elemental technologies to develop computational systems for supporting a wide range of medical services.
Task Website: http://mednlp.jp/ntcir11/

Mobile Information Access ("MobileClick")

Current web search engines usually return a ranked list of URLs in response to a query. The user often has to visit several web pages and locate relevant parts within long web pages. Especially for mobile users, those actions need much effort and attention. However, for some classes of queries, the system should be able to return a very concise summary of relevant information directly and to satisfy her immediately after clicking on the search button. In the MobileClick task, systems are expected to output a concise summary of information relevant to a given query and to provide immediate and direct information access for mobile users. For more information, please visit: http://www.dl.kuis.kyoto-u.ac.jp/ntcir-11/mobileclick/.

Recognizing Inference in TExt and Validation ("RITE-VAL")

NTCIR-11 RITE-VAL is a generic benchmark task that addresses common semantic processing needs in various NLP/Information Access research areas. We will evaluate systems which automatically detect entailment, paraphrase, and contradiction in texts written in Japanese, Simplified Chinese, Traditional Chinese, and English. RITE-VAL following RITE-1 and RITE-2 will present the main two subtasks: Fact Validation (Search) and System Validation (Unit test). In Search subtask, your system identifies whether a given text is entailed from the sentences relevant to it, which are retrieved from Wikipedia or textbooks. Unit test subtask provides a data set that includes a breakdown of linguistic phenomena that are necessary for recognizing relations between two texts. Since RITE is a friendly evaluation effort, we welcome your participation even with an ambitious/exploratory approach.
Task Website: https://sites.google.com/site/ntcir11riteval/

Spoken Query and Spoken Document Retrieval ("SpokenQuery&Doc")

Current information retrieval framework seems to face bottleneck in its human interface for drawing out one's information need. SpokenQuery&Doc task tries to overcome it by making use of spontaneously spoken queries. One of the advantage of the use of speech as an input method to retrieval systems is that it enables users to easily submit long queries to give systems rich clues for retrieval, because the unconstrained speech is common in daily use for human and the most natural and easy method to express one's thought. The target document collection is also spoken documents. Since the spoken document retrieval has been evaluated in the previous SpokenDoc-1 and 2, SpokenQuery&Doc can be considered as an extension of the previous SpokenDoc tasks, by introducing spoken queries instead of text queries.
For more information, please visit: http://www.nlp.cs.tut.ac.jp/ntcir11/



QA Lab for Entrance Exam ("QALab")

Task Website: http://ntcir.nii.ac.jp/QALab/

Temporal Information Access ("Temporalia")

Given the fact that time plays crucial role in estimating infromation relevance and validity, we believe that sussessful search engines must consider temporal aspects of information in greater detail. We set two subtasks for this purpose: Temporal Query Intent Classification (TQIC) and Temporal Information Retrieval (TIR). TQIC asks participants to classify a given query to several temporal classes. TIR asks participants to retrieve a set of documents that are relevant to information needs that incorporate time factor.
Task Website: https://sites.google.com/site/ntcirtemporalia/
Task Website (in NTCIR CMS Site): http://ntcir.nii.ac.jp/Temporalia/NTCIR-11-Temporalia/
Mailing List: https://groups.google.com/forum/#!forum/ntcirtemporalia
Facebook Page: https://www.facebook.com/ntcirtemporalia
Twitter: https://twitter.com/ntcirtemporalia

Cooking Recipe Search ("RecipeSearch")

Information access tasks involving food have traditionally focused on locating or ranking relevant restaurants given a user’s information need. However, home cooking remains a fundamental method for acquiring a meal. In this pilot, we propose exploring the information access tasks associated with food recipes. Japanese/English recipe search collections will be constructed.
Task Website: https://sites.google.com/site/ntcir11recipesearch/
Contact to Task Organizers: ntcir11.recipesearch


Last Modified: 2014-07-09