NTCIR-9 RITE Formal Run Evaluation Results
TABLE OF CONTENTS
For the updated Formal Run Evaluation Results, please see the revised version of the NTCIR-9 RITE Task Overview Paper (updated on 2012-04-24).
Evaluation result on BC subtask (JA)
| Run | Acc |
|---|
| JAIST-JA-BC-01 | 0.5800 |
| JAIST-JA-BC-02 | 0.5660 |
| JAIST-JA-BC-03 | 0.5520 |
| NTTCS-JA-BC-03 | 0.5480 |
| LTI-JA-BC-03* | 0.5460 |
| LTI-JA-BC-02* | 0.5420 |
| LTI-JA-BC-01* | 0.5340 |
| NTTCS-JA-BC-01 | 0.5320 |
| IBM-JA-BC-02* | 0.5260 |
| FX-JA-BC-02 | 0.5240 |
| FX-JA-BC-03 | 0.5200 |
| NTTCS-JA-BC-02 | 0.5200 |
| IBM-JA-BC-01* | 0.5160 |
| KYOTO-JA-BC-02 | 0.5160 |
| KYOTO-JA-BC-03 | 0.5160 |
| SITLP-JA-BC-01 | 0.5160 |
| SITLP-JA-BC-02 | 0.5120 |
| FX-JA-BC-01 | 0.5100 |
| JUCS-JA-BC-03 | 0.5080 |
| IBM-JA-BC-03* | 0.5000 |
| JUCS-JA-BC-02 | 0.5000 |
| SITLP-JA-BC-03 | 0.4940 |
| KYOTO-JA-BC-01 | 0.4920 |
| JUCS-JA-BC-01 | 0.4900 |
[TOP]
Evaluation result on BC subtask (CS)
| Run | Acc |
|---|
| UIOWA-CS-BC-01 | 0.9705 |
| UIOWA-CS-BC-03 | 0.9631 |
| UIOWA-CS-BC-02 | 0.9361 |
| ICRC_HITSZ-CS-BC-03 | 0.7764 |
| FudanNLP-CS-BC-02 | 0.7617 |
| ICRC_HITSZ-CS-BC-02 | 0.7568 |
| FudanNLP-CS-BC-01 | 0.7469 |
| WHUTE-CS-BC-03 | 0.7371 |
| NTU-CS-BC-01 | 0.7346 |
| WHUTE-CS-BC-02 | 0.7322 |
| WUST-CS-BC-01 | 0.7248 |
| NTU-CS-BC-02 | 0.7224 |
| NTU-CS-BC-03 | 0.7199 |
| ZSWSL-CS-BC-01 | 0.7199 |
| IASLD-CS-BC-01* | 0.7150 |
| ICL-CS-BC-01 | 0.7150 |
| WHUTE-CS-BC-01 | 0.7125 |
| ICL-CS-BC-02 | 0.7101 |
| ICRC_HITSZ-CS-BC-01 | 0.7076 |
| IASLD-CS-BC-02* | 0.7052 |
| IASLD-CS-BC-03* | 0.6880 |
| III_CYUT_NTHU-CS-BC-02 | 0.6830 |
| NSNG-CS-BC-02 | 0.6683 |
| ZSWSL-CS-BC-02 | 0.6658 |
| NSNG-CS-BC-01 | 0.6536 |
| Yuntech-CS-BC-01 | 0.6364 |
| NSNG-CS-BC-03 | 0.5897 |
| ZSWSL-CS-BC-03 | 0.5897 |
| Yuntech-CS-BC-02 | 0.5602 |
| III_CYUT_NTHU-CS-BC-01 | 0.5577 |
| III_CYUT_NTHU-CS-BC-03 | 0.5577 |
[TOP]
Evaluation result on BC subtask (CT)
| Run | Acc |
|---|
| UIOWA-CT-BC-01 | 0.9078 |
| UIOWA-CT-BC-02 | 0.8844 |
| IASLD-CT-BC-03 | 0.6611 |
| IASLD-CT-BC-02 | 0.6533 |
| III_CYUT_NTHU-CT-BC-02 | 0.6500 |
| IASLD-CT-BC-01 | 0.6478 |
| NTOUA-CT-BC-02* | 0.6422 |
| ICRC_HITSZ-CT-BC-01 | 0.6133 |
| NTOUA-CT-BC-01* | 0.6133 |
| NTU-CT-BC-01 | 0.6078 |
| NTU-CT-BC-03 | 0.6067 |
| NTOUA-CT-BC-03* | 0.6022 |
| ICRC_HITSZ-CT-BC-02 | 0.5967 |
| NTU-CT-BC-02 | 0.5956 |
| III_CYUT_NTHU-CT-BC-01 | 0.5733 |
| III_CYUT_NTHU-CT-BC-03 | 0.5733 |
| IMTKU-CT-BC-02 | 0.5556 |
| MCU-CT-BC-01 | 0.5544 |
| IMTKU-CT-BC-01 | 0.5500 |
| Yuntech-CT-BC-01 | 0.5278 |
| IMTKU-CT-BC-03 | 0.5244 |
| Yuntech-CT-BC-02 | 0.5244 |
[TOP]
Evaluation result on MC subtask (JA)
| Run | Acc |
|---|
| IBM-JA-MC-02* | 0.5114 |
| KYOTO-JA-MC-03 | 0.4841 |
| KYOTO-JA-MC-02 | 0.4795 |
| IBM-JA-MC-01* | 0.4545 |
| NTTCS-JA-MC-03 | 0.4523 |
| NTTCS-JA-MC-01 | 0.4477 |
| IBM-JA-MC-03* | 0.4455 |
| NTTCS-JA-MC-02 | 0.4045 |
| KYOTO-JA-MC-01 | 0.2136 |
| JUCS-JA-MC-01 | 0.1750 |
[TOP]
Evaluation result on MC subtask (CS)
| Run | Acc |
|---|
| UIOWA-CS-MC-01 | 0.8919 |
| UIOWA-CS-MC-02 | 0.8919 |
| UIOWA-CS-MC-03 | 0.8870 |
| ICRC_HITSZ-CS-MC-03 | 0.6413 |
| ICRC_HITSZ-CS-MC-02 | 0.6241 |
| ZSWSL-CS-MC-02 | 0.6192 |
| WHUTE-CS-MC-02 | 0.6093 |
| III_CYUT_NTHU-CS-MC-02 | 0.5897 |
| FudanNLP-CS-MC-02 | 0.5848 |
| WHUTE-CS-MC-01 | 0.5823 |
| WUST-CS-MC-01 | 0.5823 |
| FudanNLP-CS-MC-01 | 0.5799 |
| ICRC_HITSZ-CS-MC-01 | 0.5749 |
| NTU-CS-MC-02 | 0.5749 |
| NTU-CS-MC-03 | 0.5700 |
| IASLD-CS-MC-01* | 0.5651 |
| NTU-CS-MC-01 | 0.5651 |
| ZSWSL-CS-MC-03 | 0.5627 |
| IASLD-CS-MC-03* | 0.5553 |
| ZSWSL-CS-MC-01 | 0.5455 |
| IASLD-CS-MC-02* | 0.5430 |
| III_CYUT_NTHU-CS-MC-01 | 0.5332 |
| III_CYUT_NTHU-CS-MC-03 | 0.5307 |
| Yuntech-CS-MC-01 | 0.5283 |
| ICL-CS-MC-01 | 0.5061 |
| ICL-CS-MC-02 | 0.4840 |
| Yuntech-CS-MC-02 | 0.3980 |
[TOP]
Evaluation result on MC subtask (CT)
| Run | Acc |
|---|
| UIOWA-CT-MC-01 | 0.7867 |
| UIOWA-CT-MC-02 | 0.7744 |
| UIOWA-CT-MC-03 | 0.7244 |
| MCU-CT-MC-01 | 0.5356 |
| IMTKU-CT-MC-01 | 0.5222 |
| IMTKU-CT-MC-02 | 0.5067 |
| IASLD-CT-MC-03 | 0.5011 |
| IASLD-CT-MC-01 | 0.4989 |
| ICRC_HITSZ-CT-MC-01 | 0.4967 |
| III_CYUT_NTHU-CT-MC-02 | 0.4911 |
| IASLD-CT-MC-02 | 0.4867 |
| NTU-CT-MC-03 | 0.4833 |
| Yuntech-CT-MC-01 | 0.4767 |
| NTOUA-CT-MC-02* | 0.4611 |
| NTU-CT-MC-01 | 0.4589 |
| NTU-CT-MC-02 | 0.4578 |
| NTOUA-CT-MC-01* | 0.4400 |
| III_CYUT_NTHU-CT-MC-03 | 0.4333 |
| III_CYUT_NTHU-CT-MC-01 | 0.4300 |
| NTOUA-CT-MC-03* | 0.4211 |
| Yuntech-CT-MC-02 | 0.3878 |
| IMTKU-CT-MC-03 | 0.2678 |
[TOP]
Evaluation result on Entrance Exam subtask (JA)
| Run | Acc |
|---|
| IBM-JA-EXAM-01 | 0.7217 |
| TU-JA-EXAM-02** | 0.7183 |
| TU-JA-EXAM-03** | 0.7042 |
| IBM-JA-EXAM-02 | 0.6742 |
| LTI-JA-EXAM-03 | 0.6674 |
| KYOTO-JA-EXAM-02 | 0.6561 |
| KYOTO-JA-EXAM-03 | 0.6561 |
| LTI-JA-EXAM-02 | 0.6538 |
| JAIST-JA-EXAM-02 | 0.6516 |
| JAIST-JA-EXAM-03 | 0.6516 |
| TU-JA-EXAM-01 | 0.6493 |
| JAIST-JA-EXAM-01 | 0.6222 |
| LTI-JA-EXAM-01 | 0.6018 |
| KYOTO-JA-EXAM-01 | 0.5928 |
| IBM-JA-EXAM-03 | 0.5837 |
| JUCS-JA-EXAM-01 | 0.5204 |
| TU-JA-EXAM-02 | 0.1154 |
| TU-JA-EXAM-03 | 0.1131 |
[TOP]
Evaluation result on RITE4QA subtask (JA)
| Run | Acc | Top1 | MRR |
|---|
| LTI-JA-RITE4QA-03* | 0.6753 | 0.2136 | 0.2982 |
| JAIST-JA-RITE4QA-01 | 0.5602 | 0.1802 | 0.2765 |
| JAIST-JA-RITE4QA-03 | 0.6940 | 0.1658 | 0.2731 |
| JAIST-JA-RITE4QA-02 | 0.6763 | 0.1508 | 0.2604 |
| LTI-JA-RITE4QA-02* | 0.6411 | 0.1743 | 0.2563 |
| JUCS-JA-RITE4QA-01 | 0.5954 | 0.1315 | 0.2490 |
| KYOTO-JA-RITE4QA-02 | 0.6836 | 0.1206 | 0.2344 |
| KYOTO-JA-RITE4QA-03 | 0.6836 | 0.1206 | 0.2344 |
| IBM-JA-RITE4QA-01* | 0.3330 | 0.1131 | 0.2327 |
| IBM-JA-RITE4QA-03* | 0.4015 | 0.0871 | 0.2221 |
| LTI-JA-RITE4QA-01* | 0.8434 | 0.1265 | 0.2220 |
| IBM-JA-RITE4QA-02* | 0.3164 | 0.0905 | 0.2168 |
| KYOTO-JA-RITE4QA-01 | 0.8890 | 0.1168 | 0.1752 |
| BASELINE1 (all yes) | 0.1100 | 0.1077 | 0.1657 |
| BASELINE2 (random) | 0.5000 | 0.1025 | 0.2320 |
[TOP]
Evaluation result on RITE4QA subtask (CS)
| Run | Acc | Top1 | MRR |
|---|
| UIOWA-CS-RITE4QA-01 | 0.9010 | 0.4559 | 0.4272 |
| IMTKU-CS-RITE4QA-02 | 0.4090 | 0.2953 | 0.3998 |
| WHUTE-CS-RITE4QA-02 | 0.4876 | 0.2852 | 0.3979 |
| WHUTE-CS-RITE4QA-01 | 0.3886 | 0.2651 | 0.3773 |
| IMTKU-CS-RITE4QA-03 | 0.4716 | 0.2550 | 0.3768 |
| IMTKU-CS-RITE4QA-01 | 0.3319 | 0.2450 | 0.3744 |
| ICL-CS-RITE4QA-01 | 0.3231 | 0.2931 | 0.3545 |
| ICRC_HITSZ-CS-RITE4QA-01 | 0.6390 | 0.2479 | 0.3520 |
| WHUTE-CS-RITE4QA-03 | 0.3275 | 0.2248 | 0.3494 |
| ICRC_HITSZ-CS-RITE4QA-03 | 0.7293 | 0.2262 | 0.3398 |
| IASLD-CS-RITE4QA-01* | 0.4833 | 0.2274 | 0.3028 |
| IASLD-CS-RITE4QA-02* | 0.4803 | 0.2274 | 0.3028 |
| III_CYUT_NTHU-CS-RITE4QA-01 | 0.7525 | 0.2585 | 0.2944 |
| III_CYUT_NTHU-CS-RITE4QA-02 | 0.7162 | 0.2408 | 0.2908 |
| ICRC_HITSZ-CS-RITE4QA-02 | 0.6128 | 0.2234 | 0.2705 |
| IASLD-CS-RITE4QA-03* | 0.4352 | 0.2310 | 0.2608 |
| III_CYUT_NTHU-CS-RITE4QA-03 | 0.3377 | 0.2320 | 0.2527 |
| BASELINE1 (all yes) | 0.1906 | 0.2243 | 0.2378 |
| BASELINE2 (random) | 0.5000 | 0.2256 | 0.3566 |
[TOP]
Evaluation result on RITE4QA subtask (CT)
| Run | Acc | Top1 | MRR |
|---|
| UIOWA-CT-RITE4QA-01 | 0.9010 | 0.4559 | 0.4272 |
| IMTKU-CT-RITE4QA-03 | 0.4003 | 0.2953 | 0.3992 |
| NTOUA-CT-RITE4QA-03* | 0.6346 | 0.2813 | 0.3824 |
| NTOUA-CT-RITE4QA-01* | 0.5459 | 0.2746 | 0.3803 |
| IMTKU-CT-RITE4QA-01 | 0.3246 | 0.2517 | 0.3772 |
| IMTKU-CT-RITE4QA-02 | 0.3392 | 0.2517 | 0.3736 |
| NTOUA-CT-RITE4QA-02* | 0.5124 | 0.2282 | 0.3572 |
| ICRC_HITSZ-CT-RITE4QA-01 | 0.6390 | 0.2479 | 0.3520 |
| ICRC_HITSZ-CT-RITE4QA-03 | 0.7293 | 0.2262 | 0.3398 |
| IASLD-CT-RITE4QA-01* | 0.4760 | 0.2274 | 0.3016 |
| IASLD-CT-RITE4QA-02* | 0.4731 | 0.2274 | 0.3016 |
| III_CYUT_NTHU-CT-RITE4QA-01 | 0.7525 | 0.2598 | 0.2947 |
| III_CYUT_NTHU-CT-RITE4QA-02 | 0.7147 | 0.2408 | 0.2908 |
| ICRC_HITSZ-CT-RITE4QA-02 | 0.6128 | 0.2234 | 0.2705 |
| IASLD-CT-RITE4QA-03* | 0.4279 | 0.2290 | 0.2619 |
| III_CYUT_NTHU-CT-RITE4QA-03 | 0.3392 | 0.2320 | 0.2527 |
| BASELINE1 (all yes) | 0.1906 | 0.2243 | 0.2378 |
| BASELINE2 (random) | 0.5000 | 0.2109 | 0.3454 |
[TOP]
* IASLD, IBM, LTI, and NTOUA includes RITE organizer(s) in a team. They paid full attention to fairly participate in formal run.
** Evaluated only on pairs where a label is returned.