	DAY-1: Dec 5 (Tue)	DAY-2: Dec 6 (Wed)	DAY-3: Dec 7 (Thu)		DAY-4: Dec 8 (Fri)
	EVIA 2017	NTCIR-13	NTCIR-13		NTCIR-13
9:00			STC-2	AKG	WWW	NAILS
9:10
9:20
9:30		Welcome & NTCIR-13 Overview
9:40
9:50
10:00		Keynote (Speaker: Omar Alonso)
10:10
10:20
10:30			Coffee Break		Coffee Break
10:40					Coffee Break
10:50					OpenLiveQ	ECA
11:00		Coffee Break	STC-2	MedWeb
11:10
11:20
11:30		Task Overview: Lifelog-2, MedWeb
11:40
11:50
12:00
12:10
12:20					Poster & Lunch (QALab-3, Lifelog-2, WWW, NAILS, OpenLiveQ, ECA, NTCIR-14)
12:30		Lunch (on your own)	Break
12:40
12:50
13:00	EVIA 2017		Poster & Lunch (STC, AKG, MedWeb)
13:10
13:20
13:30
13:40
13:50					Reports from TREC and CLEF, Introduction to NTCIR-14, and Wrap Up
14:00		Task Overview: OpenLiveQ, QALab-3, AKG, ECA
14:10
14:20
14:30			Break
14:40			Break
14:50			QALab-3	Break-out Session (OpenLiveQ and AKG)
15:00
15:10
15:20
15:30
15:40
15:50			Coffee Break
16:00		Coffee Break	Coffee Break		Break-out Session (STC)
16:10			QALab-3	Lifelog-2
16:20
16:30		Task Overview: NAILS, STC-2, WWW
16:40
16:50
17:00					Break-out Session (CENTRE)
17:10			Break-out Session (QALab-3)
17:20
17:30
17:40
17:50
18:00		NTCIR 20th Anniversary Session (Speakers: Fredric C. Gey and Douglas W. Oard)			Extended Steering Committee (invited only)
18:10
18:20
18:30			Banquet
18:40
18:50

13:00 - 17:10	The 8th International Workshop on Evaluating Information Access (EVIA 2017) Co-chairs: Nicola Ferro (University of Padua) and Ian Soboroff(NIST)
13:00 - 13:10	Opening
13:10 - 13:50	EVIA Keynote Title:Cross-Language Information Retrieval in the MATERIAL Program Douglas W. Oard (University of Maryland)
13:50 - 15:00	Session 1: Evaluation Measures and Test Collections
(25 min)	Test Collections and Measures for Evaluating Customer-Helpdesk Zhaohao Zeng, Cheng Luo, Lifeng Shang, Hang Li, Tetsuya Sakai
(25 min)	An interval-like scale property for IR evaluation measures Marco Ferrante, Nicola Ferro, Silvia Pontarollo
(20 min)	Evaluating Evaluation Measures with Worst-Case Confidence Interval Widths Tetsuya Sakai
15:00 - 15:30	Coffee Break
15:30 - 17:00	Session 2: Assessment and Automatic Evaluation
(20 min)	Automatic Evaluation of World History Essay Using Chronological and Geographical Measures Kotaro Sakamoto, Hideyuki Shibuki, Madoka Ishioroshi, Akira Fujita, Yoshinobu Kano,Teruko Mitamura, Tatsunori Mori, Noriko Kando
(25 min)	Towards Automatic Evaluation of Multi-Turn Dialogues: A Task Design that Leverages Inherently Subjective Annotations Tetsuya Sakai
(25 min)	The Effect of Inter-Assessor Disagreement on IR System Evaluation: A Case Study with Lancers and Students Tetsuya Sakai
(20 min)	Unanimity-Aware Gain for Highly Subjective Assessments Tetsuya Sakai
17:00 - 17:10	Wrap-up and closing

8:45 -	Registration Open (8:45 - 17:00)
9:30 - 9:45	Welcome (at Hitotsubashi Hall ) Noriko Kando (National Institute of Informatics) Session Chairs: Makoto P. Kato (Kyoto University) and Yiqun Liu (Tsinghua University)
9:45 - 10:00	Overview of NTCIR-13 (at Hitotsubashi Hall ) Makoto P. Kato (Kyoto University) and Yiqun Liu (Tsinghua University) Session Chairs: Makoto P. Kato (Kyoto University) and Yiqun Liu (Tsinghua University)
10:00 - 11:00	Keynote (at Hitotsubashi Hall ) Title: The practice of crowdsourcing: things to know about using humans and machines for labeling Omar Alonso (Microsoft) Session Chair: (TBA)
11:00 - 11:30	Coffee Break
11:30 - 12:30	Task Overview: Lifelog-2 and MedWeb (at Hitotsubashi Hall ) Session Chairs: Makoto P. Kato (Kyoto University) and Yiqun Liu (Tsinghua University) 11:30 - 12:00 Lifelog-2: Overview of the NTCIR-13 Lifelog-2 Task 12:00 - 12:30 MedWeb: Overview of the NTCIR-13: MedWeb Task

14:00 - 16:00	Task Overview: OpenLiveQ, QALab-3, AKG, and ECA (at Hitotsubashi Hall ) Session Chairs: Makoto P. Kato (Kyoto University) and Yiqun Liu (Tsinghua University) 14:00 - 14:30 OpenLiveQ: Overview of the NTCIR-13 OpenLiveQ Task 14:30 - 15:00 QALab-3: Overview of the NTCIR-13 QA Lab-3 Task 15:00 - 15:30 AKG: Overview of NTCIR-13 Actionable Knowledge Graph (AKG) Task 15:30 - 16:00 ECA: Overview of NTCIR-13 ECA Task
16:00 - 16:30	Coffee Break
16:30 - 18:00	Task Overview: NAILS, STC-2, and WWW (at Hitotsubashi Hall) Session Chairs: Makoto P. Kato (Kyoto University) and Yiqun Liu (Tsinghua University) 16:30 - 17:00 NAILS: Overview of NTCIR-13 NAILS Task 17:00 - 17:30 STC-2: Overview of the NTCIR-13 Short Text Conversation Task 17:30 - 18:00 WWW: Overview of the NTCIR-13 We Want Web Task
18:00 - 19:00	NTCIR 20th Anniversary Session (at Hitotsubashi Hall ) Title: NTCIR from the Beginning: A Personal Research Journey Fredric C. Gey (University of California, Berkeley) Title: NTCIR in the World: Two Decades of Impact Douglas W. Oard (University of Maryland) Session Chair: (TBA)

9:00 - 9:18	Lifeng Shang, Tetsuya Sakai, Hang Li. Overview of the NTCIR-13 Short Text Conversation Task (Chinese Subtask)
9:18 - 9:36	Shih-Hung Wu, Wen-Feng Shih, Che-Cheng Yu, Liang-Pu Chen and Ping-Che Yang. CYUT-III Short Text Conversation System at NTCIR-13 STC-2 Task
9:36 - 9:54	Yihan Li, Shanshan Jiang, Lei Ding, Yixuan Tong and Bin Dong. SRCB at the NTCIR-13 STC-2 Task
9:54 - 10:12	Han Ni, Liansheng Lin and Ge Xu. Nders at NTCIR-13 Short Text Conversation 2 Task
10:12 - 10:30	Dejian Yang, Yu Wu, Zhoujun Li, Wei Wu and Can Xu. Beihang at the NTCIR-13 STC-2 Task
10:30 - 11:00	Coffee Break (30min)
11:00 - 11:15	Yunong Wu, Xin Kang, Kenji Kita and Fuji Ren. TUA1 at NTCIR-13 Short Text Conversation 2 Task
11:15 - 11:30	Haizhou Zhao, Yi Du, Hangyu Li, Qiao Qian, Hao Zhou, Minlie Huang and Jingfang Xu. SG01 at NTCIR-13 STC2
11:30 - 11:45	Hiroshi Nakatani, Shigenori Nishiumi, Takahiro Maeda and Masahiro Araki. KIT Dialogue System for NTCIR-13 STC-2 Task
11:45 - 12:00	Hiroki Tanioka. Response Generation for Grounding at NTCIR-13 STC-2
12:00 - 12:15	Toru Shimizu. YJTI at the NTCIR-13 STC Japanese Subtask

9:00 - 9:10	Roi Blanco, Hideo Joho, Adam Jatowt, Haitao Yu, and Shuhei Yamamoto. Overview of NTCIR-13 AKG Task
9:10 - 9:30	Xin Kang, Yunong Wu and Fuji Ren. TUA1 at the NTCIR-13 Action Mining Task: Sampling Related Actions from Online Searching
9:30 - 9:50	Xinshi Lin, Wai Lam and Shubham Sharma. CUIS Team for NTCIR-13 AKG Task
9:50 - 10:10	Md Mostafizur Rahman and Atsuhiro Takasu. TLAB at NTCIR-13 AKG Task
10:10 - 10:30	Discussion

11:00 - 11:15	Shoko Wakamiya, Mizuki Morita, Yoshinobu Kano, Tomoko Ohkuma and Eiji Aramaki. Overview of the NTCIR-13: MedWeb Task
11:15 - 11:30	Hayate Iso, Camille Ruiz, Taichi Murayama, Katsuya Taguchi, Ryo Takeuchi, Hideya Yamamoto, Shoko Wakamiya and Eiji Aramaki. NTCIR13 MedWeb Task: multi-label classification of tweets using an ensemble of neural networks
11:30 - 11:45	Reine Asakawa and Tomoyoshi Akiba. AKBL at the NTCIR-13 MedWeb Task
11:45 - 12:00	Anh Hang Nga Tran, Hiroko Kobayashi, Yu Sawai and Paulo Quaresma. UE and Nikon at the NTCIR-13 MedWeb Task
12:00 - 12:15	Masahito Sakishita and Yoshinobu Kano. Classification of Tweet Posters who have Diseases by Mixing Method of Rule-Based and SVM in NTCIR-13 MedWeb
12:15 - 12:25	Break
12:25 - 12:30	Chao Li, Kang Xin and Fuji Ren. Medweb Task: Identify Multi-Symptoms from Tweets Based on Active Learning and Semantic Information
12:30 - 12:35	Wei-San Lin, Hong-Jie Dai and Yu-Hsuan Shao. Principle base Approach for Classifying Tweets with Flu-related Information in NTCIR-13 MedWebTask
12:35 - 12:40	Masao Ito. NIL: Using scoring to analyse the ambiguous messages on the NTCIR-13 MedWeb task
12:40 - 12:45	Kazui Morita and Takagi Toshihisa. DrG at NTCIR-13: MedWeb Task
12:45 - 12:50	Mamoru Sakai and Hiroki Tanioka. Keyword-based Challenges at the NTCIR-13 MedWeb
12:50 - 13:00	Discussion and Closing

14:50 - 15:00	Tao-Hsing Chang, Yu-Sheng Tsai, Chih-Li Tsai and Pei-Xuan Cai. ATA2: A Question Answering System at NTCIR-13 QALab-3
15:00 - 15:10	Kou Sato and Tetsuya Sakai. SLQAL at the NTCIR-13 QA Lab-3 Task
15:10 - 15:20	Yuanzhi Ke and Masafumi Hagiwara. HagiwaraLab at the NTCIR-13 QALab-3 Task
15:20 - 15:30	Kotaro Sakamoto, Madoka Ishioroshi, Yuta Fukuhara, Akihiro Iizuka, Hideyuki Shibuki, Tatsunori Mori and Noriko Kando. Forst: Question Answering System for Second-stage Examinations at NTCIR-13 QA Lab-3 Task
15:30 - 15:40	Yusuke Doi, Takuma Takada, Takuya Matsuzaki and Satoshi Sato. SML Question-Answering System for World History Essay Exams at NTCIR-13 QALab-3
15:40 - 15:50	Tasuku Kimura, Ryo Tagami and Hisashi Miyamori. KSU Team's QA System for World History Exams at the NTCIR-13 QA Lab-3 Task
15:50 - 16:10	Coffee Break (30min)
16:10 - 16:20	Mike Tian-Jian Jiang. DGLab Question Answering System and Automatic Evaluation Method at NTCIR-13 QA Lab-3 for University Entrance Exam on World History Essay
16:20 - 16:30	Min-Yuh Day, Chao-Yu Chen, Wan-Chu Huang, I-Hsuan Huang and Shi-Ya Zheng, Tz-Rung Chen, Min-Chun Kuo, Yue-Da Lin, Yi-Jing Lin. IMTKU Question Answering System for World History Exams at NTCIR-13 QA Lab-3
16:30 - 16:40	Fadi Botros, Francesco Ciannella, Takaaki Matsumoto, Evan Chan, Cheng-Ta Chung, Lucas Bengtson, Keyang Xu, Tian Tian and Teruko Mitamura. CMUQA in QALab-3: Essay Question Answering for University Entrance Exams
16:40 - 16:50	Takaaki Matsumoto and Teruko Mitamura. MTMT in QALab-3: World History Essay Question Answering System that Utilizes Textbooks and Open Knowledge Bases
16:50 - 17:00	Tsunenori Ishioka, Kohei Yamaguchi and Tsuneori Mine. Rubric-based Automated Japanese Short-answer Scoring and Support System Applied to QALab-3

16:10 - 16:30	Review of the Lifelog-2 Task (Cathal Gurrin)
16:30 - 16:50	LSAT Sub-Task Review - VCI2R at the LSAT Sub-Task (Quanli Xu) - NTT at the LSAT Sub-Task (Yoshiaki Takimoto) - DCU at the LSAT Sub-Task (Cathal Gurrin)
16:50 - 17:00	LEST Sub-Task Review - NTT at the LEST Sub-Task (Yoshiaki Takimoto)
17:00 - 17:30	LIT Sub-Task Review - Tsinghua University at the LIT Sub-Task (TBD) - VCI2R at the LIT Sub-Task (Quanli Xu) - DCU at the LIT Sub-Task (Cathal Gurrin)
17:30 - 17:40	LAT Sub-Task Review - NTT at the LAT Sub-Task (Yoshiaki Takimoto)
17:40 - 18:10	Lifelog-3 Planning - Hideo Joho, Cathal Gurrin

14:50 - 15:50	OpenLiveQ (at NII 20F 2005) AKG (at Conference Room 2)
17:10 - 18:10	QALab-3 (at Conference Room 1)

9:00 - 9:10	Graham Healy. Overview of NTCIR-13 NAILS Task
9:10 - 9:25	Amelia Solon, Stephen Gordon, Brent Lance and Vernon Lawhern. Deep Learning Approaches for P300 Classification in Image Triage: Applications to the NAILS Task
9:25 - 9:40	Holly Hutson, Shlomo Geva and Philipp Cimiano. Ensemble Methods for the NTCIR-13 NAILS Task

11:00 - 11:15	Qinghong Gao, Hu Jiannan, Xu Ruifeng, Gui Lin, Yulan He, Kam-Fai Wong and Qin Lu. Overview of NTCIR-13
11:15 - 11:25	Xiangju Li, Shi Feng, Daling Wang and Yifei Zhang. Decision Tree Method for the NTCIR-13 ECA Task
11:25 - 11:35	Han Ren, Yafeng Ren and Jing Wan. The GDUFS System in NTCIR-13 ECA Task
11:35 - 11:45	Maofu Liu, Linxu Xia, Zhenlian Zhang and Yang Fu. WUST CRF-Based System at NTCIR-13 ECA Task
11:45 - 11:55	What is the next step of ECA?

9:00 - 9:10	Cheng Luo, Tetsuya Sakai, Yiqun Liu, Zhicheng Dou, Chenyan Xiong and Jingfang Xu. Overview of the NTCIR-13 We Want Web Task
9:10 - 9:20	Luke Gallagher, Joel Mackenzie, Rodger Benham, Ruey-Cheng Chen, Falk Scholer and J. Shane Culpepper. RMIT at the NTCIR-13 We Want Web Task
9:20 - 9:30	Ming Yue and Zhicheng Dou. RUCIR at NTCIR-13 WWW Task
9:30 - 9:40	Yukun Zheng, Cheng Luo, Weixuan Wu, Jia Chen, Yiqun Liu, Huanbo Luan, Min Zhang and Shaoping Ma. THUIR at NTCIR-13 WWW Task
9:40 - 9:50	Zhuyun Dai, Chenyan Xiong and Jamie Callan. An Evaluation of the Kernel Based Neural Ranking Model in NTCIR-13 WWW
9:50 - 10:00	Peng Xiao, Lingtao Li, Yimeng Fan and Tetsuya Sakai. SLWWW at the NTCIR-13 WWW Task
10:00 - 10:25	Discussion about future plans: Task Organizers and Participants

10:50 - 11:00	Overview of the NTCIR-13 OpenLiveQ Task
11:00 - 11:15	Tomohiro Manabe, Akiomi Nishida and Sumio Fujita. YJRS at the NTCIR-13 OpenLiveQ Task
11:15 - 11:30	Ming Chen, Lin Li, Yueqing Sun and Jie Zhang. Erler at the NTCIR-13 OpenLiveQ Task
11:30 - 11:45	Ryo Kashimura and Tetsuya Sakai. SLOLQ at the NTCIR-13 OpenLiveQ Task
11:45 - 12:00	Mengjia He, Xin Kang and Fuji Ren. TUA1 at the NTCIR-13 OpenLiveQ Task
12:00 - 12:15	Takashi Sato. OKSAT at NTCIR-13 OpenLiveQ Task
12:15 - 12:20	Wrap up

13:50 - 14:20	Reports from TREC and CLEF New Tracks for TREC 2018 Ian Soboroff (National Institute of Standards and Technology, NIST) CLEF 2018: Evaluation Labs, Conference, and Initiative Nicola Ferro (University of Padua)
14:20 - 14:55	Introduction to NTCIR-14 Session Chairs: Makoto P. Kato (Kyoto University) and Yiqun Liu (Tsinghua University) - Personal Lifelog Organisation & Retrieval (Lifelog-3) - Open Live Test for Question Retrieval (OpenLiveQ-2) - QA Lab for Politics Minutes (QA Lab-4) - Short Text Conversation (STC-3) - We Want Web (WWW-2) - CLEF/NTCIR/TREC REproducibility (CENTRE)
14:55 - 15:00	Wrap Up Makoto P. Kato (Kyoto University) and Yiqun Liu (Tsinghua University)

16:00 - 17:00	STC (at NII 20F 2005)
17:00 - 18:00	CENTRE (at NII 20F 2005)

NTCIR

The 13th NTCIR Conference

DAY-1 Dec 5 Tuesday: EVIA 2017

DAY-2 Dec 6 Wednesday: NTCIR-13 Conference (at Hitotsubashi Hall)

DAY-3 Dec 7 Thursday: NTCIR-13 Conference

9:00 - 12:30, Session A-1: NTCIR-13 Short Text Conversation Task (STC-2)

9:00 - 10:30, Session B-1: NTCIR-13 Actionable Knowledge Graph Task (AKG)

11:00 - 13:00, Session B-2: NTCIR-13 Medical Natural Language Processing for Web Document (MedWeb)

14:50 - 17:10, Session A-3: NTCIR-13 QA Lab for Entrance Exam Task (QALab-3)

16:10 - 18:10, Session B-3: NTCIR-13 Personal Lifelog Organisation & Retrieval Task (Lifelog-2)

DAY-4 Dec 8 Friday: NTCIR-13 Conference

9:00 - 10:30, Session A-4: NTCIR-13 We Want Web Task (WWW)

9:00 - 10:30, Session B-4: NTCIR-13 Neurally Augmented Image Labelling Strategies Task (NAILS)

10:50 - 12:20, Session A-5: NTCIR-13 Open Live Test for Question Retrieval (OpenLiveQ)

10:50 - 12:20, Session B-5: NTCIR-13 Emotion Cause Analysis Task (ECA)