TOOLS Home

[Japanese]

---

|Overview|
|Evaluation Metrics|Document format conversion|Relevance Assessment Management|

Tools

The followings are tools constructed by NTCIR.
For details, please click on each column in the table.

1. Overview of Data formats and Tools (mainly IR)

	Evaluation tool	run format	doc format	qrel format
NTCIR1-6	trec_eval	TREC format	ntcir_doc format (NTCIR standard document format)	Early NTCIR format
NTCIR7-	NTCIREVAL (rev. on July 12, 2023)	res format [sample]	ntcir_doc format (NTCIR standard document format)	NTCIR format [sample]
	Results transformation scripts (trec_eval format to ntcir_eval format)	IR4QA format
		TREC format
PATENT	-trec_eval for PATENT -Precision-oriented evaluation program for NTCIR-5 Passage Retrieval Task	TREC format	-	-
		*For more details about data formats and tags used, please consult README or README.html included in each Test Collection.

2. Evaluation Metrics/ scorers
Task Name Abbreviation: e.g. '7ACLIA' for 'NTCIR-7 ACLIA Test Collection'

Tool									Usage	Task	Note
ir4qa_eval.tar.gz ir4qa_eval2.tar.gz									The latest version is included in NTCIREVAL (rev. on July 12, 2023)	7ACLIA, 8ACLIA, 8CQA	-
NTCIREVAL (rev. on July 12, 2023)									-The latest version- Set of tools for computing various retrieval effectiveness metrics (Binary-relevance metrics such as Average Precision and graded-relevance metrics such as Q-measure and nDCG)	NTCIR-9	This is can be used for: NTCIR and TREC Ad Hoc Retrieval, NTCIR-8 CQA, Diversified Search
BOOTS (rev. on May 07, 2016)									BOOTS is a toolkit for conducting pairwise bootstrap tests for a given set of systems, and for computing the discriminative power.
Discpower (rev. on May 07, 2016)									Discpower is a toolkit for computing the discriminative power of evaluation measures using the randomised Tukey HSD test.
NTCIRPOOL									Set of simple shell scripts for creating pools for relevance assessments	NTCIR-9	-
RITE SDK									Java framework for Textual Entailment Recognition system development and evaluation	9RITE	-

3. Document format conversion
Task Name Abbreviation: e.g. '7ACLIA' for 'NTCIR-7 ACLIA Test Collection'

Tool									Usage	Lang.	Task
mai2ntc-r.pl									Convert Mainichi Newspaper Article Data to NTCIR standard document format	J	7.8ACLIA, 3.4.5.6CLIR, .6CLQA, 8GeoTime, 6MuST, 6OPINION, 7.8MOAT 3.4.5.6QAC,
nyt2ntc.pl									Convert New York Times Newspaper Article Data to NTCIR standard document format	E	8GeoTime, 8MOAT
xie2ntc.pl									Convert Xinhua English Newspaper Article Data to NTCIR standard document format	E	4.5CLIR, 6OPINION, 7.8MOAT
xin2ntc.pl									Convert Xinhua Simplified Chinese Newspaper Article Data to NTCIR standard document format	Cs	7ACLIA
xin2ntc.1.pl									Convert Xinhua Simplified Chinese Newspaper Article Data to NTCIR standard document format	Cs	7.8ACLIA, 7.8MOAT
yomi2ntc-clqa.pl									Convert Daily Yomiuri Newspaper Article Data to NTCIR standard document format	E	5CLIR, 5CLQA, 6OPINION
yomi2ntcir.pl									Convert Yomiuri Newspaper Article Data to NTCIR standard document format	J	4.5.6CLIR, 5CLQA, 6OPINION, 4QAC

---	Test Collection available for research purposes
---	Test Collection for the workshop Participants

4. Relevance Assessment Management

System

SEPIA

---

[Japanese] [NTCIR home]