Abstract:
Natural language understanding (NLU) has been taken as one of the core fields, and the hardest one among others, of Artificial Intelligence. As such, it has been considered an  impractical technology from the perspective of the real world application. However, the situation surrounding NLU has been changing rapidly. We have witnessed large knowledge resources being constructed, manually or automatically from huge collection of text and structured data bases in Web. We have also witnessed significant amount observation data of user behaviors being accumulated in the forms of query logs, query-click logs, networks among users, etc.  These observation data will shed (and actually are shedding) light on how linguistic expressions and their interpretation interact in  context. What was considered impractical as research scenario five years ago is now being taken as serious research and application scenario. In this talk, I will discuss how the three fields of natural language understanding, semantic-based information retrieval and knowledge management are being evolved into a single unified research field and how semantics of text and intentions manifested in user queries are to be computationally manipulated.

---

Biography:
Dr. Junichi Tsujii is a Principal Researcher of Microsoft Research Asia. He joined MSRA in May, 2011. Before joining MSRA, he was Professor of Natural Language Processing in the Department of Computer Science, University of Tokyo and Professor of Text Mining in the School of Computer Science, University of Manchester. His recent research achievements include (1) Deep semantic parsing based on feature forest model, (2) Efficient search algorithms for statistical parsing, (3) Improvement of estimator for maximum entropy model, and (4) Construction of the gold standard corpus (GENIA) for Bio Text Mining.

Dr. Tsujii has received numerous awards such as the IBM Science Award (1989), SEYMF Visiting Professorship in Hong Kong (2000), Daiwa-Adrian Prize (2004), IBM Faculty Award (2005) , Achievement Award of Japan Society for Artificial Intelligence (2008), Fellow of Information Processing Society Japan (2010) and the Medal of Honor with Purple Ribbons (2010). He was President of ACL (Association for Computational Linguistics, 2006), President of IAMT (International Association for Machine Translation (2002-2004), and President of AFNLP (Asian Association for Natural Language Processing, 2007).

---

Mark Sanderson, RMIT University, Melbourne, Australia

and
William Webber, The University of Maryland, USA

Abstract:
The standard “Cranfield” approach to the evaluation of information retrieval systems has been used and refined for nearly fifty years, and has been a key element in the development of large-scale retrieval systems. The resources created by such systematic evaluations have enabled thorough retrospective investigation of the strengths and limitations of particular variants of this evaluation approach; over the last few years, such investigation has for example led to identification of serious flaws in some experiments. Knowledge of these flaws can prevent their perpetuation into future work and informs the design of new experiments and infrastructures.
In this position statement we briefly review some aspects of evaluation and, based on our research and observations over the last decade, outline some principles on which we believe new infrastructure should rest.

---

Biography: Mark Sanderson and William Webber
Mark Sanderson is a Professor in Information Retrieval at RMIT University, Melbourne, Australia
William Webber is a researcher at University of Maryland, USA

---

　NTCIR-9 Workshop Meeting
Invited Talk 2
December 8 Thursday, 9:55-10:30
Title:  Information Retrieval Experimentation with Terrier:
Scaling Up in Size and Language Support

by

Iadh Ounis, University of Glasgow, UK

Abstract:
This talk will describe the Terrier Information Retrieval Platform, aimed at facilitating research and experimentation with large textual corpora. In particular, the talk will cover Terrier's most recent developments for supporting large-scale indexing using the MapReduce distributed framework. Additionally, it will also describe our latest efforts towards enabling retrieval in non-English languages (in particular, Asian languages), which will be a core component of the upcoming release of the platform. Overall, we hope that this talk will inform NTCIR participants on how to use Terrier for effective and efficient retrieval on Asian corpora.

---

 15:30 - 15:45
TREC/TAC
by Ian Soboroff, NIST, USA

Biography:
Ian Soboroff is Acting Group Leader at NIST in Washington DC, USA.

---

15:45 - 16:00
Medieval
by Gareth Jones, Dublin City University, Ireland

Biography:
Gareth Jones is a senior lecturer at Dublin City University, Ireland.

---

16:00 - 16:15
INEX
by Andrew Trotman, The University of Otago, New Zealand

and
Shlomo Geva, Queensland University of Technology, Australia

Biography:
Andrew Trotman is a Senior Lecturer at the University of Otago, New Zealand.
Shlomo Geva is an Associate Professor at Queensland University of Technology, Australia.

---

16:15 - 16:30
CLEF
Title: CLEF 2011: Evaluation Labs, Conference, and Initiative
by Nicola Ferro, The University of Padua, Italy

and
Hideo Joho, The University of Tsukuba.

Abstract:
Starting with 2010, a radical renewal and innovation process is taking place in the Cross Language Evaluation Forum (CLEF). CLEF became an independent event constituted by (i) Evaluation Labs, i.e. laboratories to conduct evaluation of information access systems and workshops to discuss and pilot innovative evaluation activities; (ii) a peer-reviewed Conference on a broad range of issues, including investigation continuing the activities of the Evaluation Labs; experiments using multilingual and multimodal data; in particular, but not only, data resulting from CLEF activities; and, research in evaluation methodologies and challenges. This process led to the establishment of the CLEF Initiative with a charter describing its scope and aims and new organizational structure.

This talk will thus discuss the achievements and happenings in CLEF 2011, the activities conducted in the Evaluation Labs, the discussions held during the Conference, and the new organization of CLEF.

---

Biography: Nicola Ferro and Hideo Joho
Nicola Ferro is assistant professor in Computer Science at the Department of Information Engineering and at the Faculty of Statistical Sciences of the University of Padua, Italy. His main research interests are digital libraries and archives, their architectures, interoperability, and evaluation, as well as multilingual information access and its evaluation. He is chair of the Steering Committee of CLEF (Cross Language Evaluation Forum) initiative and he coordinates the PROMISE FP7 network of excellence on experimental evaluation of multilingual and multimodal information access system. He has been program co-chair of the CLEF 2010 Conference on Multilingual and Multimodal Information Access Evaluation and served on the program committee of several conferences (among them SIGIR, ECIR, CIKM, ECDL, JCDL, WSDL). He has published more than 70 papers on digital library architectures, interoperability, and services; multilingual information access and its experimental evaluation; the management of the scientific data produced during evaluation campaigns. He is member of ACM and IEEE.

Hideo Joho is an Assistant Professor in the Faculty of Library, Information and Media Science at the University of Tsukuba.

---

16:30 - 16:45
FIRE
by Hideo Joho, The University of Tsukuba.

EVIA 2011
Panel Discussion
　December 6 Tuesday, 16:45 - 17:30
Title: TREC is 20 years old, Where Now for Evaluation Campaigns?

Session Chairs:
Mark Sanderson, RMIT University, Melbourne, Australia
and
William Webber, The University of Maryland, USA

Speakers:

Ian Soboroff (TREC/TAC)

Gareth Jones (Medieval)

Andrew Trotman and Shlomo Geva (INEX)

Hideo Joho (NTCIR)