NTCIR (NII Test Collection for IR Systems) Project Related URL'sContact InformationNII
NTCIR HOME

Search

HOME
About NTCIR
・WorkShop
NTCIR-11
NTCIR-10
NTCIR-9
NTCIR-8
NTCIR-7
NTCIR-6
NTCIR-5
NTCIR-4
NTCIR-3
NTCIR-2
NTCIR-1
Data/Tools
Publications/Online Proceedings
Related URL's
Mailing Lists
FAQ
Contact Information
PrivacyPolicy
NTCIR CMS HOME


| Test Collections| Submission Archives| Tools| User Agreements| Detailed Table of Test Collections|

NTCIR Project

Submission Archives - DATA/User Agreement
(For Research Purposes)

[Japanese]

[NTCIR home] [Test Collection]
Submission Archives consist of the results which were submitted by task participants of the NTCIR Workshop that NII organized,Evaluation Data and Evaluation Tools.

Submission Archives are the datasets to be used for research related to evaluation methods and metrics of information access systems. With regard to the purpose of use, please make contact with Noriko Kando, the NTCIR Project leader at ntc-admin@.

Class Collection Task Task Data Submission
Topic/Question Relevance Judge
Lang. # #
runs
#
groups
ACLIA NTCIR-7
ACLIA
CCLQA (Task Overview) CCLQA: QA C
E
J
CS-CS: 100
CT-CT: 100
EN-CS: 100
EN-CT: 100
EN-JA: 100
JA-JA: 100
Binary
decision
*5
total: 40
CS-CS:14
CT-CT: 3
EN-CS:10
EN-JA:6
JA-JA:7
9
Question Analysis: total: 10
CS-CS:6
CT-CT: 0
EN-CS:2
EN-JA:1

JA-JA:1
IR4QA+
CCLQA
total:
78(113)
CS-CS:
22(42)
CT-CT:0
EN-CS:
20(32)
EN-JA:
22(11)
JA-JA:
14(28)
*8
IR for QA
(Task Overview)
IR4QA IR CS-CS: 97
CT-CT: 95
EN-CS: 97
EN-CT: 95
EN-JA: 98
JA-JA: 98
3
grades
total:
91(93)
*9
CS-CS:22
CT-CT: 19
EN-CS:
18(20)
*9
EN-CT: 7
EN-JA:11
JA-JA:14
12
NTCIR-8
ACLIA
CCLQA
(Task Overview)
CCLQA QA C
E
J
CS-CS:
CT-CT:
EN-CS:
EN-CT:

EN-JA:
JA-JA:
100 for each language pair
Binary pyramid nugget matching total:19
CS-CS:6
CT-CT:3
EN-CS:1
EN-CT:3
EN-JA:3
JA-JA:3
6
Question Analysis: total:7
CS-CS:3
CT-CT:0
EN-CS:2
EN-CT:0
EN-JA:1
JA-JA:1
IR4QA+CCLQA total:21
CS-CS:0
CT-CT:3
EN-CS:0
EN-CT:9
EN-JA:4
JA-JA:5
IR for QA (Task Overview) IR4QA IR CS-CS:
CT-CT:
EN-CS:
EN-CT:
EN-JA:
JA-JA:
100* for each language pair
*2
3
grades
total:84
CS-CS:20
CT-CT:5
EN-CS:28
EN-CT:14
EN-JA:9
JA-JA:8
12
CLIR on News NTCIR-5 CLIR
(Task Overview)
IR Ct
E
J
K
50
*2
4 grades total:379
SLIR:201
BLIR:153
MLIR: 25
24
NTCIR-6 CLIR
(Task Overview)
STAGE1 IR Ct
E
J
K
50
(selected
from NTCIR-3,4)

*2
4 grades total:151
SLIR:94
BLIR: 55
MLIR: 2
20
STAGE2 total:494
*6
SLIR:234
BLIR: 260
19
CLQA NTCIR-5 CLQA
(Task Overview)
QA C
E
J
smpl:300,
test:200
*3
3
grades
*4
total:
42(47)

CC7(7)
CE7(8)
EC:8(7)
EJ:12(14)
JE:8(11)
*7
13
NTCIR-6 CLQA
(Task Overview)
QA C
E
J
C-E/C-C/E-C/E-E:
150
J-E/J-J/E-J:
200
3
grades
*4
total:
52(41)

CC14(13)
CE0(1)
EC:14(10)
EE:3(1)
EJ:12(8)
JE:1(1)
JJ:8(7)
*7
12
CQA NTCIR-8 CQA
Test Collection and Task/
System Evaluation
(Task Overview)
answer quality ranking J Questions: 1500
Answews: 7443
(Best Answers and manual assessments)
2 grades or
4 grades
10 3
INTENT NTCIR-9 INTENT
(Task Overview)

(Task Website)
Subtopic mining Cs
J
100 for each language - Cs: 42
J: 14

*15
Cs: 13
J: 5
Document Ranking Cs
J
5 grades(per Intent) Cs: 25
J: 18

*15
Cs: 7
J: 4
One Click NTCIR-9 1CLICK
(Task Overview)

(Task Website)
One Click Access J 60 2 grades
*5
10
(Open: 6
Oracle: 4)
3
Opinion NTCIR-6 OPINION
(Task Overview)

IE/
analysis
Ct 32
(selected
from NTCIR
-3,-4,-5 CLIR)
2
types,
3
metrics
opinion
ated: 7
relevance:
4
Polarity: 5
5
E opinion
ated: 9
relevance:
6
Polarity: 7
6
J opinion
ated: 5
relevance:
3
polarity: 3
3
NTCIR-7 MOAT
(Task Overview)
Cs 16 2
types,
3
metrics
opinion
ated: 18
relevance:
12
polarity: 8
opinion holder: 12
target: 5
9
Ct 17 opinion
ated: 15
relevance:
12
polarity: 12
opinion holder: 8
target: 5
7
E 17 opinion
ated: 22
relevance:
12
polarity: 14
opinion holder: 4
target: 2
9
J 22 opinion
ated: 12
relevance:
6
polarity: 8
8
NTCIR-8 MOAT
(Task Overview)
Cs 19 2
types,
3
metrics
opinion
ated: 12
relevance:
4
polarity: 6
opinion holder: 8
target: 6
6
Ct 20 opinion
ated: 15
relevance:
5
polarity: 12
opinion holder: 8
target: 6
7
E 20 opinion
ated: 18
relevance:
7
polarity: 6
opinion holder: 3
target: 2
8
J 20 opinion
ated: 7
relevance:
4
polarity: 4
3
cross-
lingual
- 5 2
Patent NTCIR-5 PATENT
Retrieval/
Classi
fication

(Task Overview)
Document Retrieval IR E
J
1,223 4
3
1
grades
84 10
Passage Retrieval IR 356 33 4
Classi-
fication
Classi-
fication
Theme
Categorization
2,008 31 4
F-term
Categorization
500 18 3
NTCIR-6 PATENT
Retrieval/
Classi
fication

(Task Overview)
English Retrieval IR E
J
3,221 3 11 5
Japanese Retrieval IR 2,908 4 22 5
Classifi-
cation
Classifi-
cation
21,606 1 43 6
Patent
Mining
NTCIR-8
PAT-MN
(Task Overview)

Subtask of Research Paper Classifi-
cation
Mining: E English 624 1 24 3
Cross-lingual
(J2E)
644 9 1
J

Japanese

644

71

4
Cross-lingual
(E2J)
624 0 0
Subtask of Technical Trend Map Creation Mining: E English 1000 1 13 4
J Japanese 1000

27

5

Patent
Tranlation
NTCIR-7
PAT-MT
(Task Overview)

Intrinsic MT E 1381 sent.
*11
E
J
Training data:
1,798,571 sent
pairs
- EJ: 24
JE: 34
15
J 1381 sent.
*12
-
Extrinsic E 124 claims 3
levels
20 12
NTCIR-8
PAT-MT
(Task Overview)

(Translation Subtask)
Intrinsic MT E 1119 sent.
*13
E
J
Training data:
3,186,284
sent
pairs
- EJ: 22 6
J 1251 sent.
*14
- JE: 21 7
Extrinsic E 91 claims 3
levels
EJ: 11 6
NTCIR-9
PatentMT
(Task Overview)

(Task Website)
C to E MT C 2000 文
*16
C-E Training data:
ca. 1 million sent
pairs
- 74 18
E to J E 2000 文
*17
E-J Training data:
3,186,284
sent
pairs
27 9
J to E J 2000 文
*18
29 12
RITE NTCIR-9 RITE
(Task Overview)

(Task Website)
Binary-class Recognizing Textual Entailment Cs
Ct
J
Cs: 407
Ct: 900
J: 500

*19
2-way
(Y/N)
Cs: 31
Ct: 22
J: 24
Cs: 12
Ct: 9
J: 8
Multi-class Cs
Ct
J
Cs: 407
Ct: 900
J: 440

*20
5-way
(F/R/B/C/I)
Cs: 27
Ct: 22
J: 10
Cs: 11
Ct: 9
J: 4
RITE4QA Cs
Ct
J
Cs: 682
Ct: 682
J: 964
2-way
(Y/N)
Cs: 17
Ct: 16
J: 13
Cs: 7
Ct: 6
J: 5
Entrance Exam J 442
*21
2-way
(Y/N)
16 6
SpokenDoc NTCIR-9 SpokenDoc
(Task Overview)

(Task Website)
Spoken Term Detection IR for Spoken Documents J CORE: 50 query terns
ALL: 50 query terns
- CORE: 13
ALL: 5
CORE:7
ALL: 2
Spoken Document Retrieval 86 query topics 2
levels
Lecture Retrieval: 11
Passage Retrieval: 10
6Lecture Retrieval: 4
Passage Retrieval: 3
WEB NTCIR-5 WEB
(Task Overview)
IR J *1 269+847 3
grades
44 6

C:Chinese (Ct:Traditional Chinese  Cs: Simplified Chinese) E:English J:Japanese K:Korean

*1: English translation is available
*2: Removed a few topics from the formal run such that a very small number of relevant document has been returned.
*3: 300+200 questions for C documents, and 300+200 questions for JE documents
*4: Right, Unsupported, Wrong
*5: system response conceptually containing the nugget or not
*6: submissions for NTCIR-3Test Collection:162, for NTCIR-4:166, for NTCIR-5:166
*7: official runs (unofficial runs)
*8: obligatory runs (optional runs)
*9: One team submitted seven EN-CS runs but the sixth and the seventh runs were not used for pooling and are excluded from the analyses.
*10: gakkai subfiles: 1997-1999, kaken subfiles: 1986-1997
*11: Reference translation (J): 1381 sentences, Human Judgement: 100 sentences * 5 runs * 3 humans
*12: Reference translation (E): 1381 sentences + 300 sentences * 2 humans, Human Judgement: 100 sentences * 15 runs * 3 humans
*13: Reference translation (J): 1119 sentences

*14: Reference translation (E):1251 sentences + 300 sentences * 3 humans
*15: Some Japanese runs were not included in the pools for the official evaluations. For more details, please see this page.
*16: Reference translation(E): 2000 sentences, Human Judgement: 100 sentences ×(adequacy: 23 runs, acceptabiity: 13runs) ×3 humans, Development Data(C-E): 2000 sentence pairs
*17: Reference translation(J): 2000 sentences, Human Judgement: 100 sentences ×(adequacy:17 runs, acceptabiity: 11runs) ×3 humans, Development Data(E-J): 2000 sentence pairs
*18: Reference translation(E): 2000 sentences, Human Judgement: 100 sentences ×(adequacy: 19 runs, acceptabiity: 14runs) ×3 humans, Development Data(J-E): 2000 sentence pairs
*19: Cs: Development Data 407 sentence pairs, Test Data 407 sentence pairs, Ct: Development Data 421sentence pairs, Test Data 900 sentence pairs, J: Development Data 500 sentence pairs, Test Data 500 sentence pairs
*20: Cs: Development Data 407 sentence pairs, Test Data 407 sentence pairs, Ct: Development Data 421 sentence pairs, Test Data 900 sentence pairs, J: Development Data 440 sentence pairs, Test Data 440 sentence pairs
*21: Development Data 449 sentence pairs, Test Data 442 sentence pairs



Research Purpose Use of Submission Archives-
In order to use the Submission Archives, please fill out the Application Form and send by E-mail to ntc-secretariat. Especially with regard to 'Purpose of Use' in the Application Form, please refer to the memorandum(Example) and inform us in detail and definitely. If you have any question or special request, please contact Noriko Kando, the NTCIR Project leader at ntc-admin@.


How to obtain Submission Archives --- The followings are the procedures to obtain the Submission Archives. The Submission Archives and data available from NII are free of charge.

  • The application form of the Submission Archives must be filled out and sent by E-mail to ntc-secretariat. -->instructions
  • After review in the NII, upon approval of your application, you will be contacted and asked to sign User Agreement (memorandum on Permission to Use) and send the form by postal mail or courier to the address below. We will inform you of more details upon approval.
  • User Agreement (memorandum on Permission to Use Test Collection):Example

    Address
        NTCIR project (Rm.1309)
        National Institute of Informatics
        2-1-2 Hitotsubashi Chiyoda-ku, Tokyo
        102-8430, JAPAN

    PHONE: +81-3-4212-2750
    FAX: +81-3-4212-2751
    Email: ntc-secretariat


Research Activities Report/Publication Report

"Research Activities Report"" and "Publication Report" should be submitted by the users of NTCIR Submission Archive.

"Research Activities Report"
The form of Research Activities Report must be filled out and sent by E-mail to ntc-report more than one month before the expiration of the Memorandum.

"Publication Report related to NTCIR"--> please refer to the page "To Publication Report related to NTCIR"



Mailing List
The release of the new test collections and correction information shall be announced through the NTCIR Mailing list: ntcir
To subscribe the Mailing list, please refer this page.

NTCIR
Please refer this page about the ongoing NTCIR.


[NTCIR home] [Test Collection]
Last modified : 2015-12-07
ntc-admin