README for Topics and Relevance Assessments of NTCIR-3 CLIR Test Collection - <Dry runs>


1. Two sets of topics

The test collection of NTCIR-3 CLIR includes two sets of topics: one is for Chinese, Japanese and English 1998-99 document sets, and another is for Korean 1994 document set. The outline is shown in the following table. (see the section 3 about document sets)

document set
(target language)
topic set
(source language)
Chinese docs
Japanese docs
English docs
Year & Language File Name
(1) 1998-99 Chinese CLIRDryRunTopic-CH98
(2) 1998-99 Japanese CLIRDryRunTopic-JA98
(3) 1998-99 Korean CLIRDryRunTopic-KR98
(4) 1998-99 English CLIRDryRunTopic-EN98
Korean docs
(5) 1994 Chinese CLIRDryRunTopic-CH94
(6) 1994 Japanese CLIRDryRunTopic-JA94
(7) 1994 Korean CLIRDryRunTopic-KR94
(8) 1994 English CLIRDryRunTopic-EN94

2. Format of Topics

<TITLE>NBA labor dispute</TITLE>
To retrieve the labor dispute between the two parties of the US National Basketball Association at the end of 1998 and the agreement that they reached.
The content of the related documents should include the causes of NBA labor dispute, the relations between the players and the management, main controversial issues of both sides, compromises after negotiation and content of the new agreement, etc. The document will be regarded as irrelevant if it only touched upon the influences of closing the court on each game of the season.
NBA (National Basketball Association), union, team, league, labor dispute, league and union, negotiation, to sign an agreement, salary, lockout, Stern, Bird Regulation.

2.2 Tags used in topics

<TOPIC> </TOPIC> The tag for each topic
<NUM> </NUM> Topic identifier
<SLANG> </SLANG> Source language code: CH, EN, JA, KR
<TLANG> </TLANG> Target language code: CH, EN, JA, KR
<TITLE> </TITLE> The concise representation of information request, which is composed of noun or noun phrase.
<DESC> </DESC> A short description of the topic.The brief description of information need, which is composed of one or two sentences.
<NARR> </NARR> A much longer description of topic.The <NARR> has to be detailed, like the further interpretation to the request and proper nouns, the list of relevant or irrelevant items, the specific requirements or limitations of relevant documents, and so on.
<CONC> </CONC> The keywords relevant

3. Combinations of document sets with each topic set

3.1 Document Sets

Topics and relevance judgment information were created for search of the following document collections. If you wish to use the topics and relevance judgments, you need to obtain the document sets.

A. 1998-99 Document sets
No. Language Name Docs
1. Chinese(C) CIRB011 (1998-99) 132,173
United Daily News (CIRB020, 1998-99) 249,508
2. Japanese(J) Mainichi Newspaper (1998-99) 220,078
3. English(E) Taiwan News and China Times English News (EIRB010, 1998-99) 10,204
Mainichi Daily News 12,723
B. 1994 Document set
4. Korean(K) Korea Economic Daily (1994) 66,146

3.2 Combinations of three components

You have to use 'correct' combinations of document sets, topic sets and relevance judgment file as follows.

A. 1998-99 Document sets
Type of run Topics Docs Relevance Judgments Files
Monolingual IR C(1998-99) C CLIRDryRunRJ-C-Rigid
J(1998-99) J CLIRDryRunRJ-J-Rigid
E(1998-99) E CLIRDryRunRJ-E-Rigid
Bilingual IR C or K or E
J CLIRDryRunRJ-J-Rigid
C or K or J
E CLIRDryRunRJ-E-Rigid
K or E or J
C CLIRDryRunRJ-C-Rigid
Multilingual IR C or J or E or K
C and J CLIRDryRunRJ-CJ-Rigid
C or J or E or K
C and E CLIRDryRunRJ-CE-Rigid
C or J or E or K
J and E CLIRDryRunRJ-JE-Rigid
C or J or E or K
C and J and E CLIRDryRunRJ-CJE-Rigid
B. 1994 Document set
Monolingual IR K(1994) K CLIRDryRunRJ-K-Rigid
Bilingual IR C or J or E
K CLIRDryRunRJ-K-Rigid

3.3 Two kinds of relevance judgment file

In this test collection, four categories of relevace are used for the judgment, i.e.,"Highly Relevant," "Relevant," "Partially Relevant," and "Irrelevant." However, since the trec_eval scoring program we use adopts bibary relevance, we have to decide the thresholds for the 4 categories of relevance. For the reason, we provide two kinds of relevance judgment file:
(1) "Rigid" relevance - "Highly Relevant" and "Relevant" are regarded as relevant.
(2) "Relaxed" relevance - "Highly Relevant", "Relevant" and "Partially Relevant" are regarded as relevant.