NTCIR-10 RITE-2 Formal Run Evaluation Results

Team	MacroF1	Accuracy	Y-F1	Y-Prec.	Y-Rec.	N-F1	N-Prec.	N-Rec.
DCUMT-JA-BC-01	80.49	81.64	75.76	84.95	68.36	85.22	79.95	91.24
WSD-JA-BC-03	80.08	80.66	76.68	77.60	75.78	83.47	82.78	84.18
SKL-JA-BC-02	79.46	79.84	76.66	74.54	78.91	82.25	84.07	80.51
BnO-JA-BC-03	78.93	79.34	75.95	74.25	77.73	81.90	83.33	80.51
WSD-JA-BC-02	78.77	79.67	74.38	78.95	70.31	83.15	80.10	86.44
WSD-JA-BC-01	78.61	79.51	74.23	78.60	70.31	82.99	80.05	86.16
SKL-JA-BC-01	78.61	78.85	76.33	71.97	81.25	80.89	85.05	77.12
BnO-JA-BC-02	78.31	79.02	74.40	76.23	72.66	82.22	80.87	83.62
BnO-JA-BC-01	77.61	78.36	73.49	75.62	71.48	81.72	80.16	83.33
KitAi-JA-BC-01	77.11	77.70	73.44	73.44	73.44	80.79	80.79	80.79
OKA1-JA-BC-02	76.71	77.05	73.88	70.71	77.34	79.53	82.42	76.84
JAIST-JA-BC-02	76.47	76.89	73.35	71.06	75.78	79.59	81.60	77.68
SKL-JA-BC-03	76.40	77.21	72.03	74.27	69.92	80.77	79.13	82.49
KitAi-JA-BC-03	76.16	76.72	72.48	71.92	73.05	79.83	80.29	79.38
JAIST-JA-BC-01	75.56	76.23	71.51	71.94	71.09	79.61	79.27	79.94
OKA1-JA-BC-01	74.59	74.59	74.30	64.55	87.50	74.88	87.83	65.25
KYOTO-JA-BC-02	74.50	75.57	69.28	73.36	65.63	79.73	76.90	82.77
IBM-JA-BC-01	74.49	74.92	71.19	68.73	73.83	77.79	80.00	75.71
IBM-JA-BC-02	73.40	73.77	70.26	67.02	73.83	76.54	79.57	73.73
JAIST-JA-BC-03	73.08	73.77	68.75	68.75	68.75	77.40	77.40	77.40
IBM-JA-BC-03	72.90	73.44	69.08	67.54	70.70	76.72	78.07	75.42
KitAi-JA-BC-02	72.35	72.46	70.63	63.92	78.91	74.07	81.63	67.80
ut12-JA-BC-01	72.23	73.93	65.36	73.89	58.59	79.11	73.96	85.03
OKA1-JA-BC-03	71.71	72.13	68.28	65.36	71.48	75.15	77.88	72.60
FLL-JA-BC-03	67.99	70.00	59.96	68.16	53.52	76.02	70.90	81.92
*TKDDI-JA-BC-03	63.83	69.02	50.13	77.24	37.11	77.53	66.94	92.09
TKDDI-JA-BC-02	63.55	68.69	49.87	76.00	37.11	77.23	66.80	91.53
*TKDDI-JA-BC-02	63.55	68.69	49.87	76.00	37.11	77.23	66.80	91.53
*TKDDI-JA-BC-01	63.45	68.69	49.60	76.42	36.72	77.29	66.74	91.81
TKDDI-JA-BC-01	63.45	68.69	49.60	76.42	36.72	77.29	66.74	91.81
FLL-JA-BC-01	63.06	68.36	49.08	75.61	36.33	77.05	66.53	91.53
Baseline-JA-BC-01	62.53	63.93	55.28	57.63	53.13	69.78	67.91	71.75
NTTD-JA-BC-03	61.90	62.30	58.03	54.45	62.11	65.77	69.50	62.43
*FLL-JA-BC-05	61.05	63.28	51.72	57.69	46.88	70.37	66.17	75.14
FLL-JA-BC-02	59.73	64.10	46.45	62.09	37.11	73.00	64.77	83.62
ut12-JA-BC-03	59.01	65.41	42.82	69.91	30.86	75.21	64.39	90.40
ut12-JA-BC-02	57.84	64.75	40.77	69.16	28.91	74.91	63.82	90.68
NTTD-JA-BC-01	57.59	64.10	40.97	66.09	29.69	74.20	63.64	88.98
*FLL-JA-BC-06	55.69	57.70	46.25	49.55	43.36	65.14	62.44	68.08
EHIME-JA-BC-01	54.34	59.34	39.22	52.63	31.25	69.46	61.57	79.66
*FLL-JA-BC-04	52.58	55.08	41.70	45.79	38.28	63.47	60.10	67.23
THK-JA-BC-01	52.40	53.28	45.92	44.65	47.27	58.87	60.18	57.63
NTTD-JA-BC-02	50.38	62.46	25.89	75.47	15.63	74.86	61.22	96.33
EHIME-JA-BC-02	50.14	51.48	41.96	42.13	41.80	58.31	58.15	58.47
JUNLP-JA-BC-01	48.83	49.02	45.72	41.32	51.17	51.93	57.34	47.46
EHIME-JA-BC-03	48.05	48.36	44.05	40.39	48.44	52.05	56.44	48.31
KYOTO-JA-BC-03	46.42	60.98	18.49	75.00	10.55	74.35	60.10	97.46
KYOTO-JA-BC-01	41.97	60.00	9.63	92.86	5.08	74.32	59.23	99.72

Team	MacroF1	Accuracy	B-F1	B-Prec.	B-Rec.	F-F1	F-Prec.	F-Rec.	C-F1	C-Prec.	C-Rec.	I-F1	I-Prec.	I-Rec.
SKL-JA-MC-01	59.96	69.53	67.18	72.13	62.86	76.47	76.85	76.10	21.15	25.58	18.03	75.06	70.54	80.19
SKL-JA-MC-02	58.25	68.61	69.29	77.19	62.86	74.94	73.36	76.59	13.59	16.67	11.48	75.17	71.49	79.25
SKL-JA-MC-03	55.45	68.07	63.24	65.15	61.43	73.85	69.70	78.54	9.20	15.38	6.56	75.51	73.33	77.83
WSD-JA-MC-03	54.39	69.53	68.29	59.57	80.00	75.29	72.73	78.05	0.00	0.00	0.00	73.99	70.51	77.83
WSD-JA-MC-02	54.18	68.98	68.83	63.10	75.71	73.95	70.67	77.56	0.00	0.00	0.00	73.94	70.04	78.30
WSD-JA-MC-01	53.98	68.80	68.29	59.57	80.00	74.70	72.48	77.07	0.00	0.00	0.00	72.93	69.36	76.89
FLL-JA-MC-01	53.67	64.96	69.01	68.06	70.00	70.35	60.56	83.90	8.82	42.86	4.92	66.50	71.35	62.26
JAIST-JA-MC-01	52.60	66.97	66.67	64.86	68.57	74.55	69.79	80.00	0.00	0.00	0.00	69.20	65.68	73.11
BnO-JA-MC-02	52.44	57.66	58.94	44.53	87.14	62.18	73.03	54.15	20.38	16.67	26.23	68.27	78.53	60.38
JAIST-JA-MC-02	52.27	65.33	63.51	60.26	67.14	71.84	59.68	90.24	5.97	33.33	3.28	67.76	80.52	58.49
BnO-JA-MC-03	52.11	65.69	59.74	54.76	65.71	72.93	67.36	79.51	5.80	25.00	3.28	69.95	69.63	70.28
BnO-JA-MC-01	52.03	66.42	65.82	59.09	74.29	72.23	67.23	78.05	0.00	0.00	0.00	70.05	68.47	71.70
JAIST-JA-MC-03	51.48	65.33	67.92	60.67	77.14	71.13	58.49	90.73	0.00	0.00	0.00	66.86	83.69	55.66
*FLL-JA-MC-04	51.27	64.23	64.94	59.52	71.43	68.84	65.78	72.20	3.17	50.00	1.64	68.15	64.56	72.17
KYOTO-JA-MC-02	50.12	64.78	59.55	49.07	75.71	69.90	69.57	70.24	0.00	0.00	0.00	71.01	67.81	74.53
*FLL-JA-MC-02	35.12	44.71	25.93	36.84	20.00	49.58	43.82	57.07	16.00	42.86	9.84	48.98	47.16	50.94
THK-JA-MC-01	30.98	49.09	21.95	75.00	12.86	60.75	47.77	83.41	28.57	52.17	19.67	43.63	54.61	36.32
Baseline-JA-MC-01	26.61	45.44	0.00	0.00	0.00	56.18	43.01	80.98	5.41	15.38	3.28	44.88	54.36	38.21
EHIME-JA-MC-03	25.89	40.33	4.76	14.29	2.86	49.46	39.26	66.83	8.82	42.86	4.92	40.51	44.38	37.26
EHIME-JA-MC-01	24.47	28.10	14.58	11.48	20.00	37.97	42.01	34.63	12.36	9.40	18.03	32.95	41.43	27.36
*FLL-JA-MC-03	22.47	34.49	11.01	15.38	8.57	40.64	34.59	49.27	0.00	0.00	0.00	38.23	37.79	38.68
EHIME-JA-MC-02	21.99	36.31	2.41	7.69	1.43	43.98	35.78	57.07	3.03	20.00	1.64	38.55	39.41	37.74
JUNLP-JA-MC-01	21.42	22.63	16.83	12.88	24.29	24.42	30.22	20.49	17.17	12.41	27.87	27.27	34.29	22.64
KYOTO-JA-MC-01	17.04	40.33	0.00	0.00	0.00	8.29	75.00	4.39	3.23	100.00	1.64	56.64	39.59	99.53

Team	MacroF1	Acc.	Correct Answer Ratio	Y-F1	Y-Prec.	Y-Rec.	N-F1	N-Prec.	N-Rec.
BnO-JA-ExamBC-02	67.15	70.31	55.56	56.96	64.71	50.87	77.34	72.76	82.55
BnO-JA-ExamBC-03	66.97	68.75	57.41	59.30	59.65	58.96	74.64	74.37	74.91
KDR-JA-ExamBC-02	66.90	68.75	51.85	59.06	59.76	58.38	74.73	74.19	75.27
BnO-JA-ExamBC-01	66.86	69.87	57.41	56.87	63.57	51.45	76.84	72.73	81.45
KDR-JA-ExamBC-03	66.64	68.30	47.22	59.20	58.86	59.54	74.09	74.36	73.82
WSD-JA-ExamBC-01	64.90	67.86	52.78	54.72	60.00	50.29	75.09	71.62	78.91
IBM-JA-ExamBC-03	64.18	64.51	45.37	60.74	53.02	71.10	67.62	76.85	60.36
WSD-JA-ExamBC-03	64.71	67.63	52.78	54.55	59.59	50.29	74.87	71.52	78.55
SKL-JA-ExamBC-02	64.04	65.63	49.07	56.50	55.25	57.80	71.59	72.66	70.55
WSD-JA-ExamBC-02	63.96	67.63	51.85	52.46	60.61	46.24	75.47	70.57	81.09
KDR-JA-ExamBC-01	63.31	64.51	49.07	56.68	53.61	60.12	69.94	72.83	67.27
SKL-JA-ExamBC-01	61.65	67.63	29.63	46.49	64.29	36.42	76.80	68.57	87.27
SKL-JA-ExamBC-03	60.47	63.17	42.59	50.15	52.53	47.98	70.80	68.97	72.73
KitAi-JA-ExamBC-01	59.84	63.17	36.11	48.28	52.74	44.51	71.40	68.21	74.91
IBM-JA-ExamBC-02	59.33	61.83	46.29	49.26	50.61	47.98	69.41	68.31	70.55
KitAi-JA-ExamBC-03	59.05	61.38	45.37	49.27	50.00	48.55	68.83	68.21	69.45
JAIST-JA-ExamBC-02	59.04	63.39	41.67	45.70	53.49	39.88	72.39	67.40	78.18
JAIST-JA-ExamBC-03	58.65	64.96	42.59	42.49	58.00	33.53	74.80	66.95	84.73
IBM-JA-ExamBC-01	58.53	61.61	44.44	47.24	50.33	44.51	69.82	67.46	72.36
JAIST-JA-ExamBC-01	57.55	63.17	40.74	42.11	53.57	34.68	73.00	66.37	81.09
KitAi-JA-ExamBC-02	57.16	58.71	39.81	49.04	46.84	51.45	65.29	67.44	63.27
KYOTO-JA-ExamBC-02	56.82	62.05	43.52	41.78	51.26	35.26	71.85	65.96	78.91
NTTD-JA-ExamBC-02	55.57	55.58	34.26	54.88	45.15	69.94	56.26	71.11	46.55
Baseline-JA-ExamBC-01	54.77	56.47	32.41	45.98	44.15	47.98	63.55	65.38	61.82
NTTD-JA-ExamBC-03	53.12	54.02	34.26	46.63	42.25	52.02	59.61	64.68	55.27
NTTD-JA-ExamBC-01	52.02	58.93	31.48	33.81	44.76	27.17	70.23	63.27	78.91
JUNLP-JA-ExamBC-01	50.46	50.89	30.56	45.81	39.91	53.76	55.10	62.79	49.09
*TKDDI-JA-ExamBC-03	49.08	62.50	28.70	22.94	55.56	14.45	75.22	63.28	92.73
TKDDI-JA-ExamBC-01	48.62	62.28	26.85	22.12	54.55	13.87	75.11	63.12	92.73
TKDDI-JA-ExamBC-02	48.62	62.28	26.85	22.12	54.55	13.87	75.11	63.12	92.73
THK-JA-ExamBC-01	43.77	62.28	26.85	11.52	61.11	6.36	76.03	62.33	97.45
KYOTO-JA-ExamBC-03	38.57	61.38	21.30	1.14	50.00	0.58	76.01	61.43	99.64
KYOTO-JA-ExamBC-01	37.86	60.94	20.37	0.00	0.00	0.00	75.73	61.21	99.27

Team	MacroF1	Acc.	Correct Answer Ratio	Search Prec.	Search Rec.	Y-F1	Y-Prec.	Y-Rec.	N-F1	N-Prec.	N-Rec.
*KDR-JA-ExamSearch-02	58.12	64.51	32.41	NA	NA	41.76	57.00	32.95	74.48	66.67	84.36
*KDR-JA-ExamSearch-01	57.59	63.84	33.33	NA	NA	41.30	55.34	32.95	73.87	66.38	83.27
*KDR-JA-ExamSearch-03	57.39	63.17	34.26	NA	NA	41.70	53.64	34.10	73.08	66.27	81.45
NTTD-JA-ExamSearch-01	55.02	58.04	25.93	17.63	43.41	43.37	45.28	41.62	66.67	65.05	68.36
*BnO-JA-ExamSearch-02	54.77	56.47	31.48	26.56	13.08	45.98	44.15	47.98	63.55	65.38	61.82
*BnO-JA-ExamSearch-01	52.45	54.91	26.85	22.32	10.99	41.62	41.62	41.62	63.27	63.27	63.27
*BnO-JA-ExamSearch-03	51.78	51.79	31.48	NA	NA	51.57	42.12	66.47	52.00	66.86	42.55
NTTD-JA-ExamSearch-02	49.15	49.33	25.93	17.63	43.41	52.21	41.06	71.68	46.08	66.44	35.27
KYOTO-JA-ExamSearch-01	46.57	62.95	28.70	47.14	3.62	17.00	62.96	9.83	76.15	62.95	96.36
KYOTO-JA-ExamSearch-02	45.41	62.28	26.85	39.29	2.42	15.08	57.69	8.67	75.75	62.56	96.00

Team	MacroF1	Accuracy	Y-F1	Y-Prec.	Y-Rec.	N-F1	N-Prec.	N-Rec.
*FLL-JA-UnitTest-01	77.77	90.87	94.84	94.39	95.28	60.71	62.96	58.62
*FLL-JA-UnitTest-03	76.98	91.29	95.13	93.61	96.70	58.82	68.18	51.72
JAIST-JA-UnitTest-02	74.52	89.21	93.87	93.87	93.87	55.17	55.17	55.17
*TKDDI-JA-UnitTest-03	74.00	85.89	91.58	96.35	87.26	56.41	44.90	75.86
*TKDDI-JA-UnitTest-01	73.51	85.48	91.32	96.34	86.79	55.70	44.00	75.86
*TKDDI-JA-UnitTest-02	73.51	85.48	91.32	96.34	86.79	55.70	44.00	75.86
TKDDI-JA-UnitTest-01	73.51	85.48	91.32	96.34	86.79	55.70	44.00	75.86
TKDDI-JA-UnitTest-02	73.51	85.48	91.32	96.34	86.79	55.70	44.00	75.86
BnO-JA-UnitTest-01	73.44	85.89	91.63	95.88	87.74	55.26	44.68	72.41
NTTD-JA-UnitTest-02	68.98	84.23	90.73	93.94	87.74	47.22	39.53	58.62
BnO-JA-UnitTest-03	68.78	80.08	87.56	97.13	79.72	50.00	35.82	82.76
JAIST-JA-UnitTest-01	67.36	79.67	87.40	96.05	80.19	47.31	34.38	75.86
NTTD-JA-UnitTest-01	61.99	74.27	83.60	95.18	74.53	40.38	28.00	72.41
THK-JA-UnitTest-01	53.26	71.37	82.35	89.94	75.94	24.18	17.74	37.93
Baseline-JA-UnitTest-01	51.70	86.31	92.58	88.41	97.17	10.81	25.00	6.90
*FLL-JA-UnitTest-02	51.35	77.59	87.08	88.35	85.85	15.63	14.29	17.24
NTTD-JA-UnitTest-03	47.91	53.94	65.63	95.50	50.00	30.19	18.46	82.76
BnO-JA-UnitTest-02	46.80	87.97	93.60	87.97	100.00	0.00	0.00	0.00
KYOTO-JA-UnitTest-02	45.35	48.96	59.41	98.90	42.45	31.28	18.67	96.55
KYOTO-JA-UnitTest-01	37.27	38.59	46.38	100.00	30.19	28.16	16.38	100.00
JAIST-JA-UnitTest-03	29.46	30.71	38.83	86.89	25.00	20.10	11.67	72.41

Team	MacroF1	Y-F1	Y-Prec.	Y-Rec.	N-F1	N-Prec.	N-Rec.
IASL-CT-BC-02	67.14	71.66	68.64	74.95	62.63	66.48	59.20
MIG-CT-BC-02	67.07	70.99	69.03	73.07	63.14	65.51	60.95
MIG-CT-BC-03	66.99	71.23	68.74	73.90	62.76	65.85	59.95
IMTKU-CT-BC-01	65.99	69.16	68.80	69.52	62.83	63.22	62.44
WHUTE-CT-BC-01	65.55	70.20	67.37	73.28	60.89	64.44	57.71
MIG-CT-BC-01	65.42	67.94	68.88	67.01	62.91	61.93	63.93
IMTKU-CT-BC-03	63.82	67.76	66.47	69.10	59.87	61.36	58.46
Yuntech-CT-BC-03	62.31	65.26	65.82	64.72	59.36	58.78	59.95
Yuntech-CT-BC-02	62.02	66.46	64.75	68.27	57.58	59.57	55.72
Yuntech-CT-BC-01	61.64	67.13	64.02	70.56	56.16	60.06	52.74
KC99-CT-BC-01	57.67	66.42	60.45	73.70	48.93	57.58	42.54
CYUT-CT-BC-01	55.16	55.77	60.14	51.98	54.55	50.75	58.96
CYUT-CT-BC-02	52.64	58.44	56.67	60.33	46.83	48.79	45.02
IASL-CT-BC-01	51.77	65.79	56.84	78.08	37.76	52.91	29.35
CYUT-CT-BC-03	51.58	63.89	56.39	73.70	39.27	50.59	32.09
JUNLP-CT-BC-01	48.72	50.82	53.20	48.64	46.63	44.47	49.00
IMTKU-CT-BC-02	48.61	36.36	63.54	25.47	60.86	48.19	82.59
NTOUA-CT-BC-01	32.63	25.06	32.34	20.46	40.20	34.08	49.00
NTOUA-CT-BC-03	31.71	19.39	28.81	14.61	44.04	35.89	56.97
NTOUA-CT-BC-02	30.70	15.20	25.37	10.86	46.20	36.83	61.94

Team	MacroF1	B-F1	B-Prec.	B-Rec.	F-F1	F-Prec.	F-Rec.	C-F1	C-Prec.	C-Rec.	I-F1	I-Prec.	I-Rec.
IASL-CT-MC-02	46.32	52.35	53.06	51.66	64.63	53.99	80.49	29.90	36.25	25.44	38.41	52.73	30.21
WHUTE-CT-MC-01	45.50	58.86	56.36	61.59	67.06	54.36	87.50	12.08	25.71	7.89	43.99	63.40	33.68
MIG-CT-MC-02	45.15	57.68	48.64	70.86	54.49	58.60	50.91	14.19	26.83	9.65	54.25	50.45	58.68
NTOUA-CT-MC-03	44.80	61.10	50.43	77.48	64.21	55.00	77.13	1.50	5.26	0.88	52.40	70.59	41.67
NTOUA-CT-MC-01	44.63	62.07	54.82	71.52	65.79	54.01	84.15	0.00	0.00	0.00	50.66	69.28	39.93
MIG-CT-MC-03	44.21	51.07	39.93	70.86	52.52	61.13	46.04	19.19	22.62	16.67	54.04	54.61	53.47
KC99-CT-MC-01	43.75	45.48	42.94	48.34	63.61	57.00	71.95	16.67	15.87	17.54	49.24	66.08	39.24
MIG-CT-MC-01	42.16	53.49	47.67	60.93	51.91	57.14	47.56	10.06	17.78	7.02	53.19	47.30	60.76
Yuntech-CT-MC-03	40.14	52.70	53.79	51.66	65.27	51.03	90.55	4.38	13.04	2.63	38.19	61.07	27.78
Yuntech-CT-MC-01	39.76	48.92	53.54	45.03	65.80	51.19	92.07	5.63	14.29	3.51	38.68	60.29	28.47
Yuntech-CT-MC-02	38.77	46.89	52.46	42.38	65.59	50.75	92.68	4.32	12.00	2.63	38.30	60.00	28.13
IMTKU-CT-MC-01	35.76	60.56	52.15	72.19	63.47	57.57	70.73	13.41	22.00	9.65	41.38	54.55	33.33
NTOUA-CT-MC-02	33.49	1.29	25.00	0.66	62.50	50.96	80.79	13.98	18.06	11.40	56.20	56.49	55.90
IASL-CT-MC-01	33.04	32.20	44.71	25.17	60.71	47.58	83.84	21.72	22.43	21.05	17.54	31.53	12.15
MCUIM-CT-MC-01	32.51	59.21	58.82	59.60	70.07	61.33	81.71	25.00	20.69	31.58	8.29	20.27	5.21
IMTKU-CT-MC-03	32.36	52.83	44.55	64.90	65.58	59.02	73.78	1.65	14.29	0.88	41.75	52.36	34.72
CYUT-CT-MC-02	26.26	21.39	18.97	24.50	43.68	38.43	50.61	17.00	15.79	18.42	22.98	38.84	16.32
CYUT-CT-MC-01	25.60	22.58	19.00	27.81	45.85	38.62	56.40	12.44	15.19	10.53	21.54	41.18	14.58
JUNLP-CT-MC-01	24.21	21.22	17.70	26.49	32.28	41.23	26.52	16.72	12.67	24.56	26.61	30.49	23.61
CYUT-CT-MC-03	23.51	27.22	19.76	43.71	45.62	39.51	53.96	0.00	0.00	0.00	21.19	41.41	14.24
IMTKU-CT-MC-02	19.37	0.00	0.00	0.00	30.63	64.08	20.12	15.95	26.53	11.40	50.26	35.79	84.38

Ranked by WorseRanking Top1 scores.

BetterRanking scores show how good a system is in terms of the improvement on the answer ranking of a good-performing factoid QA system, while WorseRanking scores show how good a system when it is applied to the answer ranking of a bad-performing factoid QA system.

We ranked system performance based on WorseRanking Top1 accuracy because a high BetterRanking score may be the result of a system that outputs most pairs with “Y” label with the same confidence score, which results in a fallback into the original QA system ranking. A baseline RITE system that always outputs “Y” with the same confidence score was created to show the effect (see orgQAsys-* runs). The answer ranking of such a system is identical to the original QA system answer ranking. In addition, according to BetterRanking scores, none of the runs performs better than the original QA ranking.

	WorseRanking						BetterRanking
CT	R			R+U			R			R+U
Run	Top1	MRR	Top5	Top1	MRR	Top5	Top1	MRR	Top5	Top1	MRR	Top5
WHUTE-CT-RITE4QA-01	27.33	34.57	46.67	30.67	38.76	52.67	26.67	34.29	46.67	30.00	38.48	52.67
IMTKU-CT-RITE4QA-03	16.67	25.70	40.67	19.33	29.39	46.00	17.33	26.03	40.67	20.00	29.72	46.00
IMTKU-CT-RITE4QA-01	14.67	22.69	37.33	22.00	31.84	49.33	14.67	22.58	37.33	22.00	31.73	49.33
CYUT-CT-RITE4QA-03	12.67	17.69	28.67	18.67	26.60	42.67	12.00	17.50	28.67	18.67	26.99	43.33
IASL-CT-RITE4QA-01	12.00	22.14	39.33	14.67	26.86	48.00	30.00	38.27	50.67	32.67	42.34	57.33
IMTKU-CT-RITE4QA-02	12.00	19.82	32.00	19.33	29.43	44.67	12.00	19.84	32.67	20.00	29.79	45.33
IASL-CT-RITE4QA-02	10.67	16.01	27.33	14.67	22.09	37.33	38.00	42.74	49.33	42.00	47.66	56.67
NTOUA-CT-RITE4QA-01	8.00	9.28	11.33	13.33	17.06	22.67	8.67	9.61	11.33	15.33	18.17	22.67
NTOUA-CT-RITE4QA-03	8.00	9.97	13.33	12.67	17.19	24.00	9.33	10.63	13.33	14.67	18.30	24.00
NTOUA-CT-RITE4QA-02	7.33	8.78	10.67	11.33	14.11	18.00	8.00	9.22	10.67	12.00	14.56	18.00
orgQAsys-CT-RITE4QA-01	7.33	11.54	22.67	10.67	16.99	31.33	40.67	47.60	57.33	44.67	52.32	64.00
CYUT-CT-RITE4QA-01	6.67	11.71	24.00	10.00	16.99	32.67	38.00	45.19	54.67	43.33	51.08	62.00
CYUT-CT-RITE4QA-02	6.67	11.71	24.00	10.00	16.99	32.67	38.00	45.19	54.67	43.33/td>	51.08	62.00

Team	MacroF1	Accuracy	Y-F1	Y-Prec.	Y-Rec.	N-F1	N-Prec.	N-Rec.
bcNLP-CS-BC-03	73.84	74.65	78.43	72.58	85.31	69.25	78.25	62.12
MIG-CS-BC-02	68.09	68.50	71.72	69.64	73.93	64.45	66.97	62.12
CYUT-CS-BC-03	67.86	68.12	70.74	70.16	71.33	64.98	65.63	64.35
bcNLP-CS-BC-01	67.04	69.65	76.32	65.98	90.52	57.75	80.20	45.13
bcNLP-CS-BC-02	66.89	69.91	76.89	65.71	92.65	56.88	83.33	43.18
MIG-CS-BC-01	65.71	65.81	67.56	69.33	65.88	63.87	62.11	65.74
CYUT-CS-BC-02	63.11	63.12	62.50	69.36	56.87	63.73	58.16	70.47
WHUTE-CS-BC-02	61.65	66.58	75.40	62.60	94.79	47.90	84.51	33.43
CYUT-CS-BC-01	61.17	61.59	57.14	71.94	47.39	65.20	55.86	78.27
*IASL-CS-BC-02	60.45	63.25	70.98	61.90	83.18	49.91	66.82	39.83
WHUTE-CS-BC-01	58.20	64.79	74.79	60.99	96.68	41.61	87.50	27.30
MIG-CS-BC-03	57.19	63.64	73.80	60.42	94.79	40.59	81.51	27.02
IMTKU-CS-BC-03	54.28	62.74	73.95	59.42	97.87	34.61	89.53	21.45
Yuntech-CS-BC-03	53.52	59.54	70.24	58.28	88.39	36.80	65.25	25.63
Yuntech-CS-BC-02	52.10	59.03	70.32	57.77	89.81	33.88	65.60	22.84
Yuntech-CS-BC-01	50.91	58.64	70.39	57.40	91.00	31.42	66.07	20.61
IMTKU-CS-BC-01	50.82	60.31	72.42	57.98	96.45	29.22	81.01	17.83
*IASL-CS-BC-01	50.60	54.03	63.63	55.58	74.41	37.57	50.00	30.08
WUST-CS-BC-02	50.14	58.77	70.89	57.31	92.89	29.39	69.07	18.66
WUST-CS-BC-01	50.14	58.77	70.89	57.31	92.89	29.39	69.07	18.66
*WUST-CS-BC-01	50.14	58.77	70.89	57.31	92.89	29.39	69.07	18.66
WUST-CS-BC-03	50.14	58.77	70.89	57.31	92.89	29.39	69.07	18.66
IMTKU-CS-BC-02	50.12	60.31	72.66	57.87	97.63	27.57	85.51	16.43
JUNLP-CS-BC-01	48.49	48.66	51.39	52.61	50.24	45.59	44.44	46.80

Team	MacroF1	Accuracy	B-F1	B-Prec.	B-Rec.	F-F1	F-Prec.	F-Rec.	C-F1	C-Prec.	C-Rec.	I-F1	I-Prec.	I-Rec.
bcNLP-CS-MC-03	56.82	61.08	66.67	77.27	58.62	67.30	53.91	89.53	38.41	64.44	27.36	54.89	69.28	45.45
*IASL-CS-MC-02	50.94	53.91	55.30	61.34	50.34	64.44	57.51	73.29	38.42	40.21	36.79	45.59	50.00	41.90
WHUTE-CS-MC-01	46.79	54.80	61.54	62.41	60.69	64.36	49.90	90.61	18.71	39.39	12.26	42.58	73.08	30.04
WHUTE-CS-MC-02	46.53	56.59	62.25	59.87	64.83	65.09	51.24	89.17	8.26	33.33	4.72	50.53	75.59	37.94
bcNLP-CS-MC-02	44.88	57.62	59.68	71.84	51.03	67.86	52.58	95.67	0.00	0.00	0.00	51.99	63.79	43.87
MIG-CS-MC-02	44.74	51.60	58.50	49.07	72.41	52.84	57.69	48.74	11.35	22.86	7.55	56.26	52.01	61.26
CYUT-CS-MC-02	42.52	48.78	53.64	51.59	55.86	56.13	48.80	66.06	12.42	18.18	9.43	47.87	55.15	42.29
MIG-CS-MC-01	41.82	49.17	56.44	50.83	63.45	50.70	57.27	45.49	5.48	10.00	3.77	54.64	47.65	64.03
Yuntech-CS-MC-02	40.91	51.22	55.02	51.83	58.62	64.81	49.90	92.42	13.43	32.14	8.49	30.40	65.79	19.76
Yuntech-CS-MC-03	40.89	51.22	53.95	51.57	56.55	65.15	50.20	92.78	13.43	32.14	8.49	31.04	63.41	20.55
WUST-CS-MC-02	40.87	52.37	59.74	56.44	63.45	62.75	47.99	90.61	3.57	33.33	1.89	37.43	71.91	25.30
WUST-CS-MC-03	40.87	52.37	59.74	56.44	63.45	62.75	47.99	90.61	3.57	33.33	1.89	37.43	71.91	25.30
*WUST-CS-MC-01	40.87	52.37	59.74	56.44	63.45	62.75	47.99	90.61	3.57	33.33	1.89	37.43	71.91	25.30
CYUT-CS-MC-01	40.37	47.63	60.34	59.33	61.38	56.72	44.94	76.90	12.31	33.33	7.55	32.12	46.62	24.51
WUST-CS-MC-01	40.33	51.73	59.31	54.65	64.83	62.20	47.86	88.81	3.57	33.33	1.89	36.26	69.66	24.51
Yuntech-CS-MC-01	40.33	50.70	53.42	50.62	56.55	64.56	49.61	92.42	13.64	34.62	8.49	29.70	63.64	19.37
CYUT-CS-MC-03	40.10	51.09	48.73	51.54	46.21	57.07	44.31	80.14	0.00	0.00	0.00	54.59	73.33	43.48
bcNLP-CS-MC-01	39.95	53.91	43.43	81.13	29.66	64.70	49.07	94.95	0.00	0.00	0.00	51.69	59.90	45.45
*IASL-CS-MC-01	34.95	41.74	37.29	48.35	30.34	59.39	47.05	80.51	25.24	26.00	24.53	17.89	28.45	13.04
MIG-CS-MC-03	34.42	43.15	53.90	39.80	83.45	58.58	51.96	67.15	12.94	13.68	12.26	12.27	70.83	6.72
IMTKU-CS-MC-03	27.26	40.20	9.81	10.83	8.97	67.10	52.67	92.42	32.14	25.86	42.45	0.00	0.00	0.00
JUNLP-CS-MC-01	24.38	24.71	22.42	19.59	26.21	27.00	32.49	23.10	22.02	16.29	33.96	26.07	32.54	21.74
IMTKU-CS-MC-01	23.89	37.64	5.85	10.00	4.14	63.20	48.73	89.89	22.82	17.71	32.08	3.69	27.78	1.98
IMTKU-CS-MC-02	19.67	36.11	7.73	12.90	5.52	57.78	41.73	93.86	8.74	10.39	7.55	4.41	31.58	2.37

Ranked by WorseRanking Top1 scores.

BetterRanking scores show how good a system is in terms of the improvement on the answer ranking of a good-performing factoid QA system, while WorseRanking scores show how good a system when it is applied to the answer ranking of a bad-performing factoid QA system.

We ranked system performance based on WorseRanking Top1 accuracy because a high BetterRanking score may be the result of a system that outputs most pairs with “Y” label with the same confidence score, which results in a fallback into the original QA system ranking. A baseline RITE system that always outputs “Y” with the same confidence score was created to show the effect (see orgQAsys-* runs). The answer ranking of such a system is identical to the original QA system answer ranking. In addition, according to BetterRanking scores, none of the runs performs better than the original QA ranking.

	WorseRanking						BetterRanking
CS	R			R+U			R			R+U
Run	Top1	MRR	Top5	Top1	MRR	Top5	Top1	MRR	Top5	Top1	MRR	Top5
IMTKU-CS-RITE4QA-03	28.00	33.77	42.67	33.33	40.87	53.33	28.00	33.77	42.67	33.33	40.87	53.33
WHUTE-CS-RITE4QA-01	18.67	27.59	43.33	22.00	33.67	54.00	18.67	27.64	43.33	22.67	34.06	54.00
IMTKU-CS-RITE4QA-02	14.67	21.44	36.00	20.00	29.08	47.33	14.67	21.44	36.00	20.00	29.08	47.33
*IASL-CS-RITE4QA-02	12.67	18.80	32.00	15.33	23.13	39.33	38.00	42.71	50.00	42.00	47.27	56.00
*IASL-CS-RITE4QA-01	12.00	22.71	42.00	15.33	27.96	50.67	31.33	38.09	48.67	36.00	43.74	56.67
IMTKU-CS-RITE4QA-01	10.67	19.91	38.67	17.33	27.21	47.33	10.67	19.91	38.67	17.33	27.21	47.33
CYUT-CS-RITE4QA-02	7.33	11.82	23.33	10.67	17.32	32.67	39.33	46.13	55.33	44.00	51.52	62.67
orgQAsys-CS-RITE4QA-01	7.33	11.32	22.00	10.67	16.99	31.33	40.67	47.60	57.33	44.67	52.32	64.00
bcNLP-CS-RITE4QA-03	6.67	8.78	11.33	7.33	9.78	12.67	8.00	9.33	10.67	9.33	10.67	12.00
CYUT-CS-RITE4QA-01	6.67	11.49	23.33	10.00	16.99	32.67	38.00	45.19	54.67	43.33	51.08	62.00
bcNLP-CS-RITE4QA-01	3.33	3.67	4.00	4.67	6.17	8.00	2.67	3.22	4.00	6.00	6.78	8.00
bcNLP-CS-RITE4QA-02	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

RITE-2 Formal Run Evaluation Results

Japanese (JA)

BC

MC

ExamBC

ExamSearch

UnitTest

Traditional Chinese (CT)

BC

MC

RITE4QA

Simplified Chinese (CS)

BC

MC

RITE4QA