Task Overview and Call for Task Participation

The 15th NTCIR (2019 - 2020)
Evaluation of Information Access Technologies
Conference: December 08-11, 2020, NII, Tokyo, Japan

Call for Participation to the NTCIR-15 Tasks:

Task Participation

Let's participate in a collaborative activity for enhancing Information Access technologies!

For the 20 years, NTCIR has been formulating the infrastructure for the evaluation, and contributing to development of the Information Access technologies. Consequently, NTCIR has been the major forum for researchers to intensively discuss the evaluation methodology of emerging information access technologies.
The 15th NTCIR, NTCIR-15, now calls for task participation of anyone interested in research on information access technologies and their evaluation, such as retrieval from a large amount of document collections, question answering and natural language processing. We welcome students, young researchers, professors who supervise students, researchers working for a company, and anyone who is interested in informatics.

The registration for NTCIR-15 Task Participation has just started.
Please visit: http://research.nii.ac.jp/ntcir/ntcir-15/howto.html

We are monitoring the COVID-19 situation closely. NTCIR-15 Conference may be go "online", "mixed (online + physical)", or "postponed" according to the situation. In any case, "online presentation" will be available at the NTCIR-15 Conference. So please do not hesitate to participate in the task(s) without fear of the travel.
For more details, please consult "ntcir-covid19@nii.ac.jp".


Evaluation Tasks

The fourteenth NTCIR (NTCIR-15) Task Selection Committee has selected the following five Core Tasks and two Pilot Tasks.
For task slides of Kick-Off event, please visit:
Slides for task introduce at the NTCIR-15 Kick-Off Event: http://research.nii.ac.jp/ntcir/ntcir-15/kickoff.html
For details and latest information, please see below and visit each task’s homepage.

DialEval-1     FinNum-2     QA Lab-PoliInfo-2     SHINRA2020-ML     WWW-3     Data Search     MART    


Dialogue Evaluation ("DialEval-1")

"Given a customer-helpdesk dialogue, estimate the quality of the entire dialogue and/or classify each customer/helpdesk turn."

DQ: Given a customer-helpdesk dialogue, return an estimated distribution of dialogue quality ratings for the entire dialogue.
ND: Given a customer-helpdesk dialogue, return an estimated distribution of labels over nugget types (similar to dialogue acts) for each turn.
Data: Chinese and English
For more information, please visit: http://sakailab.com/dialeval1/

Website: http://sakailab.com/dialeval1/


Numeral Attachment in Financial Tweets("FinNum-2")

"Numeral attachment in financial tweets"

In order to understanding the numeral information in depth, we proposed a task in NTCIR-14 to disambiguate the meaning of the numerals in financial social media data. However, only understanding the meanings of numerals is not enough for practical uses, because there may has more than one cashtag in a financial tweet. Understanding their semantic roles in the financial social media data is needed when mining fine-grained opinions toward a certain target.
Along this line, we design another novel task for fine-grained numeral understanding in financial social media data, called numeral attachment, which aims to detect the attached target (i.e., cashtag) of the numeral. That is, we attempt to understand that the numeral is attached to which cashtag in a tweet. For example, there are two cashtags and one numeral in (T1). The numeral "36.50" is related to $BEXP, instead of $KOG.
(T1) $KOG Took a small position- hopefully a better outcome than getting kneecapped by $BEXP selling itself dirt cheap at 36.50

Website: https://sites.google.com/nlg.csie.ntu.edu.tw/finnum2020


Question Answering Lab for Political Information ("QA Lab-PoliInfo-2")

"Generating a summary of speeches in Japanese local assembly minutes, classifying a political stance of a speaker, and identifying a mention in a speech as the same real world entity"

We propose the QA Lab-PoliInfo-2 (Question Answering Lab for Political Information 2) task at NTCIR 15 is aimed to verify the credibility of political information including fake news, using complex real-world question answering (QA) technologies.

Website: http://poliinfo2.net/
Contact: qalab-admin


SHINRA 2020 Multi-lingual("SHINRA2020-ML")

"Categorizing 30 language Wikipedia into Extended Named Entity categories"

Based on manually categorized Japanese Wikipedia pages and the language links between Wikipedia pages in different languages, the task is to categorize the entire 30 language Wikipedia pages into Extended Named Entity. We expect the participants to use the Wikipedia pages with the link from Japanese Wikipedia as the training data, and run the system to categorize the remaining Wikipedia pages which don't have language links. After the task is over, we (including the participants) will combine the results by all the participants (i.e. by Ensemble learning), and publish the results to the public. It is a scheme called "Resource by Collaborative Contribution (RbCC)" and we are expecting many participants with a good will.

Website: http://shinra-project.info/shinra2020ml/?lang=en


We Want Web with CENTRE ("WWW-3")

"Adhoc web search, with optional replicability and reproducibility runs"

The Chinese subtask is an adhoc web search task where a large query log data can be utilised. The English subtask is an adhoc web search task where not only regular adhoc runs but also replicated/reproduced runs are evaluated. All runs are required to process both the WWW-2 topics and the new WWW-3 topics.

Website: http://sakailab.com/www3/



Data Search ("Data Search")

"Ad-hoc retrieval for statistical data"

The Data Search task focuses on the retrieval of a statistical data collection published by the Japanese government (e-Stat), and one published by the US government (Data.gov).

Website: https://ntcir.datasearch.jp/


Micro Activity Retrieval Task ("MART")

"Detecting microactivities in a rich multi-modal lifelogging sensor streams"

The NTCIR15-MAD task aims to motivate the development of a first generation of techniques for high-precision micro-activity detection and retrieval of micro-activities of daily living, to support identification and retrieval of activities that occur over short time-scales, such as minutes, rather than the long-duration event segmentation tasks of the past work. Participating researchers will develop and benchmark approaches to retrieve micro-activities from rich time-aligned multi-modal sensor data. The chosen sensors will capture a lifelog camera data stream, biosignal activity (EEG, EOG, GSR, HR) and computer accesses to record interaction with digital artefacts.

Website: http://ntcir-mart.computing.dcu.ie/


Last Modified: 2020-06-04