NTCIR-11 MobileClick Evaluation Results

[NTCIR Home]

For details, please refer to the MobileClick website and the Overview of the NTCIR-11 MobileClick Task in the NTCIR Online Proceedings.

Queries

• MC1-E-Queries.tsv (English)
• MC1-J-Queries.tsv (Japanese)

This file contains test queries for English or Japanese MobileClick task. Each line is of the following format:

<QueryID>\t<Query>

e.g.

MC-E-0001 \t marlon brando acting style

iUnits

• MC1-E-iUnits.tsv (English)
• MC1-J-iUnits.tsv (Japanese)

This file contains a list of iUnits for each query. Each line consists of a query ID, an iUnit for the query, and a knowledge source of the iUnit. The format is:

<QueryID>\t<iUnit>\t<Score>\t<Source>

• <Score> is the score of the iUnit by which the iUnits are sorted in descending order.
• <Source> is the knowledge sources from which the iUnit was extracted and the filename of a baseline search result.

iUnit Retrieval Results

• eval_results.dep_ignore.tsv (dependencies were ignored)
• eval_results.dep_anywhere.tsv (iUnit X depending on iUnit Y is relevant only if Y appears in the result list)

This file contains scores of evaluation metrics for each query. Each line is of the following format:

<QueryID>\t<TeamID>\t<RunID>\t<#Retrieved>(\t<Score>)*

where <Score> is either nDCG@k or Q@k (see our overview paper).

iUnit Summarization Results

• SUM-results-140.tsv
• SUM-results-280.tsv
• SUM-results-560.tsv
• SUM-results-840.tsv

This file contains M-measure for each query. Each line is of the following format:

<RunfileName>\t<QueryID>\t<M>

where \t is a TAB, and <M> is a M-measure score (see our overview paper).

L in the filename is a patience parameter used in M-measure, which we set to 280, 560, and 840.

[NTCIR Home]