README for Topics and Relevance Assessments of NTCIR-3 CLIR Test Collection - <Formal runs>

Apr./01/2003

1. Two sets of topics

The test collection of NTCIR-3 CLIR includes two sets of topics: one is for Chinese, Japanese and English 1998-99 document sets, and another is for Korean 1994 document set. The outline is shown in the following table. (see the section 3 about document sets)

document sets
(target language)
topic sets
(source language)
1998-99
Chinese docs
Japanese docs
English docs
Year & Language File Name
(1) 1998-99 Chinese CLIRFormalRunTopic-CH98
(2) 1998-99 Japanese CLIRFormalRunTopic-JA98
CLIRFormalRunTopic-JA98r
(*see below)
(3) 1998-99 Korean CLIRFormalRunTopic-KR98
(4) 1998-99 English CLIRFormalRunTopic-EN98
1994
Korean docs
(5) 1994 Chinese CLIRFormalRunTopic-CH94
(6) 1994 Japanese CLIRFormalRunTopic-JA94
(7) 1994 Korean CLIRFormalRunTopic-KR94
(8) 1994 English CLIRFormalRunTopic-EN94

* NOTE: There is a mistranslation in Japanese topic set for 1998 named as "CLIRFormalRunTopic-JA98.*" in the preliminary version for NTCIR WS 3 participants. The topic number of the topic which contains the error is "010". It had been found in the process of relevance assessments and corrected, then the relevance assessments were carried out for the modified topic.
The error had not been found in the process of the search result submissions from the participants and the process of pooling the documents from the results yet. Therefore the pooling for the topic "010" was not enough effective, since some of the searches for it might have failed. It might be one of the reasons why the relevance assessments might not be exhaustive.
"NTCIR-3 CLIR test collection for the research purpose" includes both of the preliminary version of the topic set and the revised (error-corrected) version. The revised Japanese topic set are named "CLIRFormalRunTopic-JA98r.*".
If you use the topic "010" for your experiment, please pay attention to this point.

2. Format of Topics

2.1A sample of topics

<TOPIC>
<NUM>013</NUM>
<SLANG>CH</SLANG>
<TLANG>EN</TLANG>
<TITLE>NBA labor dispute</TITLE>
<DESC>
To retrieve the labor dispute between the two parties of the US National Basketball Association at the end of 1998 and the agreement that they reached.
</DESC>
<NARR>
The content of the related documents should include the causes of NBA labor dispute, the relations between the players and the management, main controversial issues of both sides, compromises after negotiation and content of the new agreement, etc. The document will be regarded as irrelevant if it only touched upon the influences of closing the court on each game of the season.
</NARR>
<CONC>
NBA (National Basketball Association), union, team, league, labor dispute, league and union, negotiation, to sign an agreement, salary, lockout, Stern, Bird Regulation.
</CONC>
</TOPIC>

2.2 Tags used in topics

The tags used in topic are shown as follows.

<TOPIC> </TOPIC> The tag for each topic
<NUM> </NUM> Topic identifier
<SLANG> </SLANG> Source language code: CH, EN, JA, KR
<TLANG> </TLANG> Target language code: CH, EN, JA, KR
<TITLE> </TITLE> The concise representation of information request, which is composed of noun or noun phrase.
<DESC> </DESC> A short description of the topic.The brief description of information need, which is composed of one or two sentences.
<NARR> </NARR> A much longer description of topic.The <NARR> has to be detailed, like the further interpretation to the request and proper nouns, the list of relevant or irrelevant items, the specific requirements or limitations of relevant documents, and so on.
<CONC> </CONC> The keywords relevant

3. Combinations of document sets with each topic set

3.1 Document Sets

Topics and relevance judgment information were created for search of the following document collections. If you wish to use the topics and relevance judgments, you need to obtain the document sets.

A. 1998-99 Document sets
No. Language Name Docs
1. Chinese(C) CIRB011 (1998-99) 132,173
United Daily News (CIRB020, 1998-99) 249,508
2. Japanese(J) Mainichi Newspaper (1998-99) 220,078
3. English(E) Taiwan News and China Times English News (EIRB010, 1998-99) 10,204
Mainichi Daily News 12,723
B. 1994 Document set
4. Korean(K) Korea Economic Daily (1994) 66,146

3.2 Combinations of three components

You have to use 'correct' combinations of document sets, topic sets and relevance judgment file as follows.

A. 1998-99 Document sets
Type of run Topics Docs Relevance Judgments Files
Monolingual IR C(1998-99) C CLIRFormalRunRJ-C-Rigid
CLIRFormalRunRJ-C-Relax
J(1998-99) J CLIRFormalRunRJ-J-Rigid
CLIRFormalRunRJ-J-Relax
E(1998-99) E CLIRFormalRunRJ-E-Rigid
CLIRFormalRunRJ-E-Relax
Bilingual IR C or K or E
(1998-99)
J CLIRFormalRunRJ-J-Rigid
CLIRFormalRunRJ-J-Relax
C or K or J
(1998-99)
E CLIRFormalRunRJ-E-Rigid
CLIRFormalRunRJ-E-Relax
K or E or J
(1998-99)
C CLIRFormalRunRJ-C-Rigid
CLIRFormalRunRJ-C-Relax
Multilingual IR C or J or E or K
(1998-99)
C and J CLIRFormalRunRJ-CJ-Rigid
CLIRFormalRunRJ-CJ-Relax
C or J or E or K
(1998-99)
C and E CLIRFormalRunRJ-CE-Rigid
CLIRFormalRunRJ-CE-Relax
C or J or E or K
(1998-99)
J and E CLIRFormalRunRJ-JE-Rigid
CLIRFormalRunRJ-JE-Relax
C or J or E or K
(1998-99)
C and J and E CLIRFormalRunRJ-CJE-Rigid
CLIRFormalRunRJ-CJE-Relax
B. 1994 Document set
Monolingual IR K(1994) K CLIRFormalRunRJ-K-Rigid
CLIRFormalRunRJ-K-Relax
Bilingual IR C or J or E
(1994)
K CLIRFormalRunRJ-K-Rigid
CLIRFormalRunRJ-K-Relax

3.3 Two kinds of relevance judgment file

In this test collection, four categories of relevance are used for the judgment, i.e.,"Highly Relevant," "Relevant," "Partially Relevant," and "Irrelevant." However, since the trec_eval scoring program we use adopts binary relevance, we have to decide the thresholds for the 4 categories of relevance. For the reason, we provide two kinds of relevance judgment file:
(1) "Rigid" relevance - "Highly Relevant" and "Relevant" are regarded as relevant.
(2) "Relaxed" relevance - "Highly Relevant", "Relevant" and "Partially Relevant" are regarded as relevant.

4. The sets of topics used for each sub-task

4.1 Overview

The NTCIR-3 CLIR task has many sub-tasks of which combinations of languages used in CLIR task are different each other. It should be carefully noted that the topic set to be used is determined by which languages are employed in the target 'document' set, not depending on the language used in 'topics.'

4.2 Searching the Chinese document set (C)

The Chinese (C) document set contains 381,681 Chinese documents.
The set of 42 topics (1998-99) should be used for retrieval experiments using the C collection:
Topic No.1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 32, 33, 34, 35, 36, 37, 38, 39, 40, 42, 43, 45, 46, 47, 48, 49, and 50.

4.3 Searching the Japanese document set (J)

The Japanese (J) document set contains 220,078 Japanese documents.
The set of 42 topics (for 1998-99) should be used for retrieval experiments using the J collection:
Topic No. 2, 4, 5, 7, 8, 10, 12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, and 50.

4.4 Searching the English document set (E)

The English (E) document set contains 22,927 English documents.
The set of 32 topics (for 1998-99) should be used for retrieval experiments using the E collection:
Topic No. 2, 4, 5, 7, 9, 12, 13, 14, 18, 19, 20, 21, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39, 42, 43, 45, 46, and 50.

4.5 Searching the Chinese and Japanese document set (CJ)

The Chinese (C) and Japanese (J) document set contains 601,759.
All 50 topics (for 1998-99) should be used for retrieval experiments using the CJ collection.

4.6 Searching the Chinese and English document set (CE)

The Chinese (C) and English (E) document set contains 404,608 Chinese and English documents. The set of 46 topics (for 1998-99) should be used for retrieval experiments using the CE collection:
Topic No. 1 ,2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 31, 32 ,33, 34, 35, 36, 37, 38, 39, 40, 42, 43, 45, 46, 47, 48, 49, and 50.

4.7 Searching the Japanese and English document set (JE)

The Japanese (J) and English (E) document set contains 243,005 Japanese and English documents. The set of 45 topics (for 1998-99) should be used for retrieval experiments using the JE collection: Topic No. 2, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, and 50.

4.8 Searching the Chinese, Japanese and English document set (CJE)

The Chinese, Japanese and English document set (CJE) document set contains 624,686 Chinese, Japanese, and English documents.
All 50 topics (for 1998-99) should be used for retrieval experiments using the CJE collection.