[JAPANESE] [NTCIR Home] [NTCIR DATA Home]

NTCIR-4 CLIR(IR Test Collection)

The NTCIR-4 CLIR test collection can be used for experiments of cross-lingual information retrieval between Chinese(traditional), Japanese, Korean and English(CJKE) such as

• Multilingual CLIR (MLIR)
• Bilingual CLIR (BLIR)
• Single Language (Monolingual) IR (SLIR).

*It should be noted that document sets provided for MLIR are only CJE and CJKE..

The documents are full-text of news articles in CJKE languages, which were published in Asian areas from 1998 to 1999. The test collection also includes 60 search topics in CJKE, and relevance judgement information.

＊The entire collection is provided by NII.

(1) List of document sets

The documents sets included in the NTCIR-4 CLIR test collection are as follows.

(2) Tags used in document records

The format of each news article is consistent by using a set of tags.

(1)List of topic files

The test collection of NTCIR-4 CLIR includes a set of topics for Chinese, Japanese Korean and English. The file names are as follows:

(2) A sample of topics

(3) Tags used in topics

The tags used in topic are shown as follows.

Relevance Judgments

(1) List of judgment files

(2) Two kinds of relevance judgment file

In this test collection, four categories of relevance are used for the judgment, i.e.,"Highly Relevant," "Relevant," "Partially Relevant," and "Irrelevant." However, since the trec_eval scoring program we use adopts binary relevance, we have to decide the thresholds for the 4 categories of relevance. For the reason, we provide two kinds of relevance judgment file:
(a) "Rigid" relevance - "Highly Relevant" and "Relevant" are regarded as relevant.
(b) "Relaxed" relevance - "Highly Relevant", "Relevant" and "Partially Relevant" are regarded as relevant.

(3) Format of Relevance Judgement File

The format of a relevance judgment file is as follows:

• Fields are separated by a TAB
• The list was sorted by topic ID in ascending order, then by document ID in ascending order. A "topic-ID" is the three-digit number of the search topic.
• The "dummy" filed contains information on the four categories judgment. i.e., S - "Highly Relevant," A - "Relevant ," B - "Partially Relevant," and C - "Irrelevant."
• The "relevance judgement" filed contains a binary value: "1" means "Relevant," and "0" represents "Irrelevant," which converted according to "Rigid" and "Relaxed" criteria, mentioned above.
• The "comment" field contains phrases or passages showing the reason why the documents were judged as relevant or partially relevant were extracted from the document by each assessor.

If the number of relevant documents is over 1000, the upper limit of average precision computed by the trec_eval scoring program is less than 1.0. For example, when the number of relevant documents is 1072, the upper limit seems to be 1000/1072 (= 0.9328).

The test collection has been constructed and used for the NTCIR. They are usable only for the research purpose use.
The documents collection included in the test collection were provided to NII for used in NTCIR free of charge or for a fee. The providers of the document data kindly understand the importance of the test collection in the research on information access technologies and then granted the use of the data for research purpose. Please remember that the document data in the NTCIR test collection is copyrighted and has commercial value as data. It is important for our continued reliable and good relationship with the data producers/providers that we researchers must behave as a reliable partners and use the data only for research purpose under the user agreement and use them carefully not to violate any rights for them .

The followings are the procedures to obtain the test collection. The test collection and data available from NII are free of charge.

Task　data ( without document data )

• NTCIR-4 CLIR Task data are downloadable from NII/IDR at;
  http://www.nii.ac.jp/dsc/idr/en/ntcir/ntcir.html

Document data

Please read "The terms of use" carefully before you apply data.

• 1. Please fill out The application form of the test collection and sent by E-mail to ntc-secretariat as an attachment file.
  
• 2. After review of your application in NII,we will contact you the result within a few days.
  
  < If your application are accepted >
  Please send us "User Agreement (memorandum on Permission to Use Test Collection)" by postal mail in the format shown in the following.
  
  • Please download and make two copies of the form in double-sided print.
  • Please fill out and sign on both agreement forms. (Signatures are needed on both agreement forms. )
  • Please send them by postal mail or courier to the Address below.
    
  
  
• 3. After we received agreement forms, we will distribute the data.
  
  After being counter-signed by the NII side, one copy of the User Agreement Form will be sent to you and one copy will be kept by NII.
  Because it shows permission to use the data, please keep it in a safe place during the term available.
  

Documents to submit

• Application Form [txt]
• The User Agreement (sent by email)

Reference

• The terms of use [PDF]
  

Address

NTCIR Project (Rm.1309)
National Institute of Informatics
2-1-2 Hitotsubashi Chiyoda-ku, Tokyo
102-8430, JAPAN

PHONE: +81-3-4212-2750
FAX: +81-3-4212-2751
Email: ntc-secretariat

Mailing List

The release of the new test collections and correction information shall be announced through the ntcir Mailing list

• ---

  [JAPANESE] [NTCIR Home] [Top of this page] [NTCIR DATA Home]
  Updated on : 2015-07-22
  

  ntc-admin