----------------------------------------------------------------
                 Evaluation Results of the formal Run
                         Translation Subtask
                        (Extrinsic Evaluation)
                    NTCIR-8 Patent Translation Task
                              2010.3.30


A. Extrinsic evaluation

A-0. BLEU score
     We measured BLEU of submitted files using original Japanese topic
     sentences as the reference. The computing procedure is the same as
     for EJ translation of the intrinsic evaluation.

A-1. Evaluation Method

(1) Each of the submitted translation result was used as an input for
    a Japanese monolingual patent retrieval system. Please see the
    following paper for details of that system.

     Atsushi Fujii and Tetsuya Ishikawa.
     Document Structure Analysis for the NTCIR-5 Patent Retrieval Task.
     Proceedings of the Fifth NTCIR Workshop Meeting on Evaluation of
     Information Access Technologies: Information Retrieval, Question
     Answering and Cross-lingual Information Access, pp.292-296, 2005.
     http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings5/data/PATENT/NTCIR5-PATENT-FujiiA.pdf

(2) TREC_EVAL (version 8.1) was used to calculate a number of evaluation
     measures for each submission, from which MAP (Mean Average Precision)
     and Recall@N (N=100,200,500,1000) were used for evaluation purposes.
     Recall@N is the recall for the first N documents in the retrieval result.

     Please see the following URL for details of TREC_EVAL.
     http://trec.nist.gov/trec_eval/

B. Files

  * ntc8patmt-MT-fmlrun-extrinsic-result.xls

    This excel file consists of the following two sheets.

    - extrinsic (system description & BLEU):
                 The system description and BLEU values of each run.

    - extrinsic (MAP & Recall@N):
                 While the column D ("all") denotes the results, averaged
                 over all the topics, the columns E-CQ ("3361"-"5033")
                 denote the results on a topic-by-topic basis.

    "JPN1" and "MOSES1" denote the results for the monolingual IR and
    the baseline system produced by Moses. In JPN1, the source topic
    file in Japanese (i.e., "ntc8-patmt-fmlrun-ext-ref.txt" above) was
    used as the input.

----------------------------------------------------------------

NTCIR-8 Patent Translation Task Organizers
ntcadm-patmt@cl.cs.titech.ac.jp