NTCIR Project
NTCIR-11 Temporalia
Research Purpose Use of Test Collection


NTCIR-11 Temporalia (foster research in temporal information access)

The NTCIR-11 Temporal Information Access (Temporalia) Task is to foster research in temporal information access.
Given the fact that time plays a crucial role in estimating information relevance and validity we believe that successful search engines must consider temporal aspects of information in greater detail.
At NTCIR-11, we build test collections for temporal query intent classification (TQIC) and temporal information retrieval (TIR).

Collection Document Data  Task Data
Corpus Language Volume Year Subtask Type   Language Volume Year File name
NTCIR-11 Temporalia Living Knowledge Corpus 2011-2013 *(A) English ca. 3.8M docs (ca. 20GB) Collected during May 2011 and Feb 2013 TQIC Query English 300 queries 2014 "NTCIR-11_TQIC_Queries_with_Class.xml"
TIR Document English 50 topics / 4,578 docs - Topic data: "NTCIR-11_TIR_Topics.xml"
relevance judgement: "NTCIR-11_TIR_Relevance_Judgements.tsv"

*(A): Document Data is not distributed from NII. Users are required to obtain it by themselves.
- Please see here.

Living Knowledge Corpus

Please read http://ntcirtemporalia.github.io/NTCIR-11/collection.html


For each query the following information is available

<query_string>the difference between mass and weight</query_string>
<query_issue_time>May 1, 2013 GMT+0</query_issue_time>

The "id" refers to a unique query identifier in the collection.
The "query_string" is the query and the "query_issue_time" is the query submitting date.
The "temporal_class" is the answer class for a query.


For each topic the following



<title>Trip to Bali</title>

<description>Bali has been a popular tourist destination for ...</description>

<query_issue_time>Mar 30, 2013 GMT+0:00</query_issue_time>


<subtopic id="039a" type="atemporal">What makes Bali a popular tourist destination for vacation?</subtopic>

<subtopic id="039p" type="past">What dramatic events happened in Bali before 2010?</subtopic>

<subtopic id="039r" type="recency">What are some recent news items concerning a vacation in Bali?</subtopic>

<subtopic id="039f" type="future">What are future touristic developments in Bali?</subtopic>



The "id" refers to a unique query identifier in the collection.
The "title" is brief description of search intents.
The "description" is background and motivation of search.
The "query_issue_time" is the date when the search was conducted.
The "subtopic" is specific question to answer by searching.
The attribute of "type" and "id" in "subtopic" tags are the temporal class and identifier in each question.

relevance judgement

For relevance judgement of each document the following

r Document ID Rel
001a lk-20110830040101_530 L0
001a lk-20111001040102_2640 L1
001a lk-20111005040101_3110 L2

The "r" column is identifier of each subtopic. The last letter represents a temporal class (a=atemporal, f=future, r=recency, p=past)
The "Document ID" column is document identifier in Living Knowledge Corpus.
The "Rel" column is the relevance assessment value for the topic (L0=Not Relevant, L1=Relevant, L2=Highly Relevant)


The following is the procedure to obtain the test collection. The test collection and data are available from NII free of charge.


Contact us: ntc-secretariat


The test collection was constructed and used for the NTCIR project. It is usable only for research purposes.
The document collection included in the test collection was made available to NII for use in the NTCIR project free of charge or for a fee. The providers of the document data understand the importance of such test collections in research on information access technologies and have kindly given their permission to use the data for research purposes. Please remember that the document data in the NTCIR test collection is copyrighted and has commercial value as data. To maintain a good relationship with the data producers/provider, we researchers must be reliable partners and use the data only for research purposes under the user agreement, and we must use the data carefully so as not to violate copyright.