NTCIR Project
NTCIR-9 Crosslink
(Cross-lingual Link Discovery)
Research Purpose Use of Test Collection


NTCIR-9 CROSSLINK (Cross-lingual Link Discovery)

NTCIR-9 Crosslink test collection can be used for experiments of cross-lingual link discovery from English to CJK (Chinese, Japanese and Korean) document linking such as

For each subtask, English documents are provided as topics; and for each topic it is required to identify prospective anchors and recommend links for them in the CJK document collections.

Collection Document Data Task Data
Document Type Language Year Volume Topic Relevance judgment
Language #

Wikipedia articles



318,736 English Training 3 one set qrel (from Wikipedia ground-truth )@for training topics



Korean 2010 201,596 Test 25 two set of qrels (from Wikipedia ground-truth and manual assessment) for test-topics

NTCIR-9 Crosslink Document Collections (CJK XML Corpora) and Topics are distributed under the conditions of Creative Commons Attribution-Share-Alike License 3.0 (Unported).

For more details, please visit this page:

NTCIR-9 Crosslink Document Collections are available at:

The documents sets included in the NTCIR-9 CROSSLINK test collection are formed of search engine friendly xml files created from Wikipedia mysql database dumps taken on June 2010. The original article text containing unique Wikipedia mark-ups is converted into XML using the YAWN system [1]. The details of the collections are given in the following Table.
Language # doc Size Dump Date
Chinese 318736 2.7G 27/06/2010
Japanese 716,088 6.1G 24/06/2010
Korean 201596 1.2G 28/06/2010


Most tags are kept as the same as the tags in original Wikipedia XML dump.
Some important tags are given below:

Mandatory tags



The tag for document title



The tag "id" of first level is the document identifier



The tag for link including general link and language link.
Language link contains special attribute, e.g. "xlink:label="ko"". The language code : zh, ja, ko, en



Last update timestamp



including a list of sub-categories

Other tags



Paragraph marker



Section identifier. An article often includes multiple sections.

Training and Test Topics are available at:

* All Topics are downloadable from NII/IDR. Please see here.

Training and Test Topics
Training topics

# Title ID
1 Australia 4689264
2 Femme fatale 299098
3 Martial arts 19501

Only three topics are used for system training in the NTCIR 9 Crosslink task.

Test topics
A set of 25 articles will be randomly chosen from the English Wikipedia and
used as formal test topics.

Both training and test topics can be used to generate evaluation run for system benchmarking. Please note that these topics should be orphaned by removing all hyperlinks to and from these documents by participants. The corresponding topic pages in Chinese, Japanese and Korean should also be removed from document collections.


<!ELEMENT crosslink-submission (details, description, collections, topic+)>
<!ATTLIST crosslink-submission participant-id CDATA #REQUIRED run-id CDATA #REQUIRED task (A2F| A2B) #REQUIRED default_lang (zh|ja|ko) ) #REQUIRED>
<!ELEMENT details (machine, time)>
<!ELEMENT machine (cpu, speed, cores, hyperthreads, memory)>
<!ELEMENT speed (#PCDATA)>
<!ELEMENT cores (#PCDATA)> <!ELEMENT hyperthreads (#PCDATA)> <!ELEMENT memory (#PCDATA)> <!ELEMENT time (#PCDATA)> <!ELEMENT description (#PCDATA)> <!ELEMENT collections (collection+)> <!ELEMENT collection (#PCDATA)> <!ELEMENT topic (outgoing)>
<!ELEMENT outgoing (anchor+)>
<!ELEMENT anchor (tofile+)>
<!ELEMENT tofile (#PCDATA)>
<!ATTLIST tofile bep_offset CDATA #REQUIRED lang (zh|ja|ko)#REQUIRED title CDATA #REQUIRED >

The root element crosslink-submission should contain information about participant's ID, run ID (which should include your university affiliation), the task which should be either A2F or A2B and the default target language (zh, ja, or ko). The linking algorithm should be described in description node. The collections element contains a list of document collections used in the run. Generally, the collection element should contain text from one of the following: Chinese Wikipedia, Japanese Wikipedia or Korean Wikipedia. Each topic should be contained in a topic element which should contain an anchor element for each anchor-text that should be linked. Each anchor element should include offset, length and name attributes for detailed information of the recommended anchor, and should also have one or more tofile sub-elements with the target document ID contained within them. The tofile element should contain following information: language id, title and bep (specified in lang, title, and bep_offset attributes separately) of the linked document.

Example submission
<crosslink-submission participant-id="QUT"

         <cpu>Intel Celeron</cpu>

      <time>3.04 seconds</time>

   <description>Describe the approach here, NOT in the run-id.</description>
      <collection>Chinese Wikipedia</collection>

   <topic file="9638" name=" 99 Luftballons">
   <anchor offset="768" length="8" name="Balloons">
      <tofile bep_offset="637" lang=hzhh  title=hŸƒ‹…h >4424</tofile>
      <tofile bep_offset=g238343h lang=hzhh  title=hŸƒ‹…ŕy?h >442489</tofile>

      <tofile bep_offset=g23438h lang=hzhh  title=hŸƒ‹…’j›wŽ–Œh >64424</tofile>
      <tofile bep_offset=g8997h lang=hzhh  title=h?Ÿƒ‹…”V—ˇh >14424 </tofile>
      <tofile bep_offset=g334h lang=hzhh  title=h”L—^Ÿƒ‹…h >43224</tofile>


For more details about the submission format, please visit this page.

To obtain Gold Standard (qrels, Relevance Judgment Data), please visit here.

An example Gold Standard
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
	<ltw_Topic name="Australia" id="4689264"> 
<outLink>457369</outLink> <outLink>479260</outLink> </outgoingLinks> </ltw_Topic> </ltwResultsetType>

The toolkits for assessment and evaluation in the NTCIR-9 Crosslink task are available at:


The test collection and data are available from NII free of charge.

Contact us : ntc-secretariat

Use and/or redistribution of the NTCIR-9 Crosslink CJK XML Corpora and Topics is permitted under the conditions of Creative Commons Attribution-Share-Alike License 3.0(Unported).
Details can be found at http://creativecommons.org/licenses/by-sa/3.0/.

[JAPANESE] [NTCIR Home] [Top of this page] [NTCIR Data Home]
Updated on : 2015-07-24