NTCIR

The 13th NTCIR (2016 - 2017)
Evaluation of Information Access Technologies
Conference: December 5-8, 2017, NII, Tokyo, Japan

Call for Participation to the NTCIR-13 Tasks:

Call for Participation [Flyer]
Task Participation

NTCIR Aims: http://research.nii.ac.jp/ntcir/ntcir-13/aims.html

The thirteenth NTCIR (NTCIR-13) Task Selection Committee has selected the following five Core Tasks and four Pilot Tasks.
For task slides of Kick-Off event, please visit:
Slides for task introduce at the NTCIR-13 Kick-Off Event: http://research.nii.ac.jp/ntcir/ntcir-13/kickoff.html
For details and latest information, please see below and visit each task’s homepage.

Lifelog-2    MedWeb    OpenLiveQ    QALab-3    STC-2    AKG    ECA    NAILS    WWW    

CORE TASKS

Personal Lifelog Organisation & Retrieval Task ("Lifelog-2")

"Information access and enrichment task for rich multimedia lifelog data of real-world life experience of individuals"

Abstract:
Personal lifelogging is the process of capturing multiple aspects of one's life in digital form, and is set to become an increasingly important aspect of people's lives. The main objectives of this Lifelog-2 task are to build upon the momentum of the NTCIR-12 lifelog task, to develop a new (more semantically rich) test collection, and to encourage collaborative research in this field within the wider multimedia analytics and IR communities by organising useful and novel research challenges. To this end, we are proposing the continuation of two sub-tasks from NTCIR-12 (LSAT and LIT) and include one new task (LET). The LSAT task will continue to advance community capabilities for accessing lifelogs by exploring search challenges, the LIT task will take a novel and attractive view of lifelog insights generation, while the LET task focuses on exploring approaches to semantic enrichment of raw lifelog data.

Website: http://ntcir-lifelog.computing.dcu.ie/
Contact:

PAGE TOP

Medical Natural Language Processing for Web Document ("MedWeb")

Labeling and extracting disease names of Twitter message texts and disease journal texts.

Abstract:
Recently, increasing number of medical records has been written as the form of electronic media replacing paper media, thereby digital information processing in medical fields has been increasingly needed. Nowadays, this trend focuses not only on electronic health records but also on various patients’ texts, such as social media texts, web blogs, and so on. In the NTCIR-13 MedWeb Task, participants are required to extract disease information from two types of patient texts in Japanese, English, and Chinese; (1) Twitter message texts and (2) disease journal texts. Since this task setting can be formalized as labeling or extracting disease names of these medical texts, the achievements of this task can be almost directly applied to a fundamental engine for actual applications.

Website: http://mednlp.jp/medweb/NTCIR-13/
Contact:

PAGE TOP

Open Live Test for Question Retrieval ("OpenLiveQ")

Question retrieval task in which participants can evaluate their systems in a production environment of Yahoo Japan Corporation’s community question-answering service.

Abstract:
Open Live Test for Question Retrieval (OpenLiveQ) task provides an open live test environment of Yahoo Japan Corporation’s community question-answering service for question retrieval systems. The task aims to provide an opportunity for a more realistic evaluation, to encourage participants to address question retrieval problems specific to a production environment (e.g. ambiguous/underspecified queries and diverse relevance criteria). The task is simply defined as follows: given a query and a set of questions with their answers, return a ranked list of questions.

Website: http://www.openliveq.net/
Contact:

PAGE TOP

QA Lab for Entrance Exam ("QALab-3")

Queston analysis, information retrieval, information extraction and automatic summarization using Japanese university entrance exams of "World History"

The goal is to investigate the real-world complex Question Answering (QA) technologies using Japanese university entrance exams and their English translation on the subject of "World History". The questions were selected from two different stages - The National Center Test for University Admissions (multiple choice-type questions) and from secondary exams at multiple universities (complex questions including essays). All the questions are provided in an XML format.
Website: http://research.nii.ac.jp/qalab/
Contact: qalab-admin

PAGE TOP

Short Text Conversation("STC-2")

"Given a microblog post, retrieve or generate a coherent and useful response. "

Abstract:
At NTCIR-12, STC is taken as an IR problem by maintaining a large repository of post-comment pairs from Weibo in Chinese subtask and Twitter in Japanese subtask, and then finding a clever way to reuse these existing comments to respond to new posts. At NTCIR-13, besides the retrieval- based method, we propose to add a new subtask called generation-based method to generating “new” comments. The generation-based method has emerged as a hot research topic and gained the most attention in recent years, while it is still an open problem whether the IR-based method should be wholly replaced by or combined with generation-based method for STC task. By organizing this task at NTCIR-13, we will provide a transparent platform to compare the two aforementioned methods via doing comprehensive evaluations. Furthermore, participants are encouraged to explore some effective ways to combine the two methods to get a more intelligent chatbot in the future runs of STC.

Website:
http://ntcirstc.noahlab.com.hk/STC2/stc-cn.htm
Contact:

PAGE TOP

PILOT TASK

Actionable Knowledge Graph ("AKG")

"Mining potential actions of users, and predicting the properties of search queries."

Abstract:
Knowledge graph has become increasing common and important component in search engine result pages (SERPs). However, the current knowledge graph remains informational and do not work with complex queries. This task aims to facilitate the development of new technologies to create “actionable” (or transactional) knowledge graph presentation that can be used for query intents such as buying event tickets, listening to music online or reserving a restaurant. In this pilot task, we set two subtasks: Action mining task and actionable knowledge graph generation task.

Website: http://ntcirakg.github.io/index.html
Contact:

PAGE TOP

Emotion Cause Analysis ("ECA")

"Explore the cause of emotion from text."

Abstract:
In recent years, there has been a large body of research work on emotion classification from text. However, in many cases, we care more about the stimuli, or the cause of emotions. Unfortunately, the lack of annotated corpora and standard metrics for this task has limited the research on this topic. Thus, we propose to organize a new task, emotion cause analysis (ECA), in NTCIR 13 to evaluate the state-of-the-art emotion cause extraction/detection methods on Chinese and English news text. We set two subtasks: 1.detecting the clause which contains emotion cause, and 2. Detecting the exact boundary of the emotion cause.

Website: http://hlt.hitsz.edu.cn/ECA.html
Contact:

PAGE TOP

Neurally Augmented Image Labelling Strategies ("NAILS")

"Evaluating the potential of using neural signals to label image content"
Abstract:
EEG (Electroencephalography) has recently become an accessible method for researchers and users to build and operate BCI (Brain-Computer Interface) applications, primarily because of the availability of a new generation of low-cost devices. While the initial use of such techniques began in clinical/rehabilitative settings for the purposes of augmenting communication and control, a recent trend has been to use such signals and methods in new domains, such as the image annotation task, which relies on the identification of target brain events to trigger labeling.
This trend is particularly relevant to the IR community as in recent years EEG has become a promising technology for the purposes of annotating multimedia content, or identifying when a user’s attention is drawn to something in the real world, or even as a source of user sensor data to be indexed for later retrieval. However, working with EEG data is challenging and contains many pitfalls for inexperienced researchers.
Our task will explore better ways to annotate a collection of multimedia data, introduce IR researchers to EEG data, allow them to explore the application of EEG data in a controlled experimental way, and produce the first EEG in IR test collection.
Website: http://ntcir-nails.computing.dcu.ie/
Contact:

PAGE TOP

We Want Web ("WWW")

"Ad hoc web search is back - we will monitor progress over at least three rounds of NTCIR."
Abstract:
We would like to run an ad hoc Web task for at least three rounds at NTCIR, and evaluate systems not only with traditional measures but also with more advanced measures that better reflect user experiences. For some of our test topics, we will provide user behaviour data to participants so that they can tune their systems for the latter type of measures. Participants will try to go beyond the IR performance plateau; We organisers will establish statistically-motivated methods for designing and maintaining test collections and for quantifying the progress across NTCIR rounds. Both parties will collectively conduct failure analysis for the next NTCIR rounds. WWW would use the up-to-date Sogou-Q-16(querylog) and Sogou-T-16(corpus) to promote researches.
Website: http://www.thuir.cn/ntcirwww
Contact:

PAGE TOP