NTCIR (NII Test Collection for IR Systems) Project Related URL'sContact InformationNII
NTCIR HOME

Search

HOME
About NTCIR
・WorkShop
NTCIR-11
NTCIR-10
NTCIR-9
NTCIR-8
NTCIR-7
NTCIR-6
NTCIR-5
NTCIR-4
NTCIR-3
NTCIR-2
NTCIR-1
Data/Tools
Publications/Online Proceedings
Related URL's
Mailing Lists
FAQ
Contact Information
PrivacyPolicy
NTCIR CMS HOME


| Test Collections| Submission Archives| Tools| User Agreements| Detailed Table of Test Collections|

NTCIR Project

TOOLS Home

[Japanese]

|Overview|
|Evaluation Metrics|Document format conversion|Relevance Assessment Management|

Tools

The followings are tools constructed by NTCIR.
For details, please click on each column in the table
.

1. Overview of Data formats and Tools (mainly IR)
Evaluation tool run format doc format qrel format
NTCIR1-6 trec_eval TREC format ntcir_doc format
(NTCIR standard document format)
Early NTCIR format
NTCIR7- NTCIREVAL (rev. on Oct 17, 2016) res format
[sample]
ntcir_doc format
(NTCIR standard document format)
NTCIR format
[sample]
Results transformation scripts
(trec_eval format to ntcir_eval format)
IR4QA format
TREC format
PATENT -trec_eval for PATENT

-Precision-oriented evaluation program for NTCIR-5 Passage Retrieval Task
TREC format - -
*For more details about data formats and tags used, please consult README or README.html included in each Test Collection.
*The following Test Collections contain the evaluation metrics and scorers:
NTCIR-5 CLQA, NTCIR-6 CLQA, NTCIR-3 QAC, NTCIR-4 QAC, NTCIR-5 QAC


2. Evaluation Metrics/ scorers
Task Name Abbreviation: e.g. '7ACLIA' for 'NTCIR-7 ACLIA Test Collection'

Tool

Usage

Task

Note
ir4qa_eval.tar.gz
ir4qa_eval2.tar.gz
The latest version is included in NTCIREVAL (rev. on Oct 17, 2016) 7ACLIA, 8ACLIA, 8CQA -
NTCIREVAL (rev. on Oct 17, 2016) -The latest version-
Set of tools for computing various retrieval effectiveness metrics
(Binary-relevance metrics such as Average Precision and graded-relevance metrics such as Q-measure and nDCG)
NTCIR-9 This is can be used for:
NTCIR and TREC Ad Hoc Retrieval, NTCIR-8 CQA, Diversified Search
BOOTS (rev. on May 07, 2016) BOOTS is a toolkit for conducting pairwise bootstrap tests for a given set of systems, and for computing the discriminative power.     
Discpower (rev. on May 07, 2016) Discpower is a toolkit for computing the discriminative power of evaluation measures using the randomised Tukey HSD test.     
NTCIRPOOL Set of simple shell scripts for creating pools for relevance assessments NTCIR-9 -
RITE SDK Java framework for Textual Entailment Recognition system development and evaluation 9RITE -


3. Document format conversion
Task Name Abbreviation: e.g. '7ACLIA' for 'NTCIR-7 ACLIA Test Collection'

Tool

Usage Lang.

Task

mai2ntc-r.pl Convert Mainichi Newspaper Article Data to NTCIR standard document format J 7.8ACLIA, 3.4.5.6CLIR, .6CLQA, 8GeoTime, 6MuST, 6OPINION, 7.8MOAT 3.4.5.6QAC,
nyt2ntc.pl
Convert New York Times Newspaper Article Data to NTCIR standard document format E 8GeoTime, 8MOAT
xie2ntc.pl Convert Xinhua English Newspaper Article Data to NTCIR standard document format E 4.5CLIR, 6OPINION, 7.8MOAT
xin2ntc.pl Convert Xinhua Simplified Chinese Newspaper Article Data to NTCIR standard document format Cs 7ACLIA
xin2ntc.1.pl Convert Xinhua Simplified Chinese Newspaper Article Data to NTCIR standard document format Cs 7.8ACLIA, 7.8MOAT
yomi2ntc-clqa.pl Convert Daily Yomiuri Newspaper Article Data to NTCIR standard document format E 5CLIR, 5CLQA, 6OPINION
yomi2ntcir.pl Convert Yomiuri Newspaper Article Data to NTCIR standard document format J 4.5.6CLIR, 5CLQA, 6OPINION, 4QAC
--- Test Collection available for research purposes
--- Test Collection for the workshop Participants

4. Relevance Assessment Management
System
SEPIA



[Japanese] [NTCIR home]    
Last modified : 2016-10-19
ntc-admin