README

NTCIR-3 Workshop
Spoken Query for Speech-Driven Web retrieval

Last modified: 
2003/04/30     initial version

Outline

* This task was a subtask of NTCIR-3 Web Retrieval task.
It was planned to development a test collection in order to
improve the speech-driven IR technology.

* Speech data which is collected by reading queries of text IR is used
to evaluate from a viewpoint of speech recognition and Web retrieval.

* The relevance judgement and the evaluation method are based on
NTCIR-3 Web Retrieval task.

* In NTCIR-3 Web Retrieval task, a pooling was performed for relevance
judgement. However, any result of the speech-driven IR task wasn't
used in the pooling.

Data

* Spoken Query (Dryrun)		see README.dryrun
* Spoken Query (Formal run)	see README.formalrun
* N-gram Word Language Model	see README.lm

Evaluation

Since the main purpose of this task is not that all teams compete
under the same condition, a participant can evaluate by the individual
viewpoint freely. However, in order to enable the effective discussion
for the improvement in technical of speech-driven IR, the common
measure for evaluating a technique and a system is defined and used in
the summary of the task.

* Word errer rate for speech recognition
* Term errer rate for speech recognition
* Computation time for speech recognition
* Precision and recall for Web retrieval

Organizer

* Katunobu ITOU (Nagoya Univ.)
itou@is.nagoya-u.ac.jp

* Atsushi Fujii (Univ. of Tsukuba)
http://www.slis.tsukuba.ac.jp/~fujii/jindex.html
fujii@slis.tsukuba.ac.jp