-----------------------------------------------------------------------------
                  Evaluation Results of the Formal Run
                 NTCIR-9 Patent Machine Translation Task
                               2011.12.6
                  (This readme file is encoded in UTF-8)


File:

  ntc9patmt-fmlrun-intrinsic.{CE|JE|EJ}.xlsx:
        This file contains system information, automatic evaluation results, 
        and a summary of human evaluation results (adequacy and acceptability).
        The main evaluation results are human evaluation.



Automatic Evaluation Procedure:

    We measured BLEU, NIST, and RIBES of the submitted files for the intrinsic 
    automatic evaluation using a single reference. RIBES is the score based on Isozaki et al. (2010).

    Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada,
    Automatic Evaluation of Translation Quality for Distant Language Pairs, EMNLP 2010.

  Tools

    The tools used for the automatic evaluation.

    For all subtasks

        NIST's mteval-v13a.pl
        NTT's RIBES.py version 1.01
        http://www.kecl.ntt.co.jp/icl/lirg/ribes/index.html

    For EJ subtask

        perl module Lingua::JA::Regular::Unicode version 0.05
        Mecab: version 0.98
        http://sourceforge.net/projects/mecab/files/
        Dictionary for Mecab: mecab-ipadic-2.7.0-20070801.tar.gz
        http://sourceforge.net/projects/mecab/files/mecab-ipadic/
        nkf: version 2.1.1
        http://sourceforge.jp/projects/nkf/downloads/48945/nkf-2.1.1.tar.gz/

  Procedure for CE and JE subtasks

    We used "mteval-v13a.pl" for tokenization and calculation of the BLEU and NIST scores.
    We used the "mteval-v13a.pl" tokenization function for tokenization, and "RIBES.py" for calculation of the RIBES score.
    The scores are case sensitive. The default parameters of the tools, except for case sensitivity, were used. 

  Procedure for EJ subtask

    We used the following tokenization:
    (1) Removing all white spaces (single-byte) from the submitted files.
    (2) Converting single-byte letters, numbers, and special symbols into multibyte characters for standardization purposes.
    (3) Tokenizing all Japanese sentences by Mecab (version 0.98) with mecab-ipadic-2.7.0-20070801. We concatenated a sequence of arabic numerals into a word.

    More specifically, we used the following procedure of standardization and tokenization, and used "mteval-v13a.pl" for the BLEU and NIST scores, and "RIBES.py" for the RIBES score. The default parameters of the tools were used.

    Procedure of standardization and tokenization for Japanese sentences
    cat j.txt | \
    perl -pe 's/ +//g;' | \
    perl -MEncode -MLingua::JA::Regular::Unicode -ne 'if (s/[ \n\x80-\xff]+//){ print $&; } while (s/[\x00-\x7f]+//) { print Encode::encode_utf8(alnum_h2z($&)); if (s/[ \n\x80-\xff]+//) { print $&; } }' | \
    nkf -We | \
    mecab -O wakati | \
    nkf -Ew | \
    perl -Mencoding=utf8 -pe 'while(s/([０-９]) ([０-９])/$1$2/g){} s/ $//;' > j.tok.txt

-----------------------------------------------------------------------------
NTCIR-9 Patent Machine Translation Task Organizers