Dialogue Quality and Nugget Detection (DQND) Subtasks

Recently, many reserachers are trying to build automatic helpdesk systems. However, there are very few methods to evaluate such systems.
In STC-3 NDDQ subtasks, we aim to explore methods to evaluate customer-helpdesk dialogues automatically. This dataset have the following features:

• Chinese customer-helpdesk dialogues carwled from Weibo.

• English dialgoues: manually translated from a subset of the Chinese dialgoues.

• Nugget type annotatoins for each turn: indicate whether the current turn is useful to accomplish the task.

• Quality annotation for each dialogue.

  • task accomplishment
  • customer satisfcation
  • dialogue effectiveness

In NTCIR-14 STC3-NDDQ, we consider annotations ground truth, and participants are required to predict nugget type for each turn (Nugget Detection, or ND) and dialogue quality for each dialogue (Dialogue Quality, or DQ).

The data collection of STC3-DQND consists of the following files:

• train_data_cn.json: Chinese Training data
• train_data_en.json: English Training data
• test_data_cn: Chinese Training data
• test_data_en: English Training data
• eval.py: Evaluation script.
• readme.md

For more details, please refer to the readme.md file and Overview of the NTCIR-14 Short Text Conversation Task:Dialogue Quality and Nugget Detection Subtasks in NTCIR-14 online proceedings.

The test collection of STC3-DQND is available for research purpose only:

Reference

• The terms of use [PDF]
• NTCIR-14 STC-3 CECG Task Overview : Overview of the NTCIR-14 STC-3 CECG Task [PDF]
• NTCIR-14 STC-3 DQND Task Overview : Overview of the NTCIR-14 STC-3 DQND Task [PDF]
• NTCIR-14 STC-3 website

Contact us: ntc-secretariat