NTCIR-14 Conference

The 9th International Workshop on Evaluating Information Access (EVIA 2019)

EVIA 2019

The Ninth International Workshop on Evaluating Information Access (EVIA 2019),
a Satellite Workshop of the NTCIR-14 Conference
June 10, 2019 at National Institute of Informatics, Tokyo, Japan.

EVIA 2019 Online Proceedings (released June 10, 2019)

EVIA 2019 Program -NTCIR-14 DAY-1- (June 10, 2019)

**EVIA 2019 submission was closed.**

EVIA 2019 CALL FOR PAPERS   Submission Instructions   Important Dates  
EVIA 2019 Program Committee   Past EVIAs


We invite submissions for the Ninth International Workshop on Evaluating Information Access (EVIA 2019) which will be held in conjunction with NTCIR 14 in Tokyo, Japan, on June 10, 2019.

Information Access technologies provide the interface between human information needs and digital information resources. The reliable evaluation of these technologies has been recognized for decades as central to the advancement of the field. As information retrieval technologies become more pervasive, the forms of retrieval more diverse, and retrieval tools richer, the importance of effective, efficient, and innovative evaluation grows as well.

We invite both short papers (2-4 pages) and long papers (8-10 pages) addressing one or more of the following topics, as well as any other topic related to the evaluation of information access:

- Test collection formation, evaluation metrics, and evaluation environments
- Statistical issues in information retrieval evaluation
- User studies and the evaluation of human-computer interaction in information retrieval (HCIR)
- Evaluation methods for multilingual, multimedia, or mobile information access
- Novel information access tasks and their evaluation
- Evaluation and assessment using implicit user feedback, crowdsourcing, living labs, or inferential methods
- Evaluation issues in industrial and enterprise retrieval systems
- Reproducibility issues in information retrieval evaluation

Furthermore, in this year, we specifically encourage researches on evaluation of conversational search systems, that is, information access systems that work with the user over a session to explore one or more search needs, with back-and-forth interaction between the user and the system (beyond traditional approaches like query suggestions and relevance feedback).

Accepted papers will be included in the EVIA 2019 proceedings in the CEUR Workshop Proceedings (CEUR-WS.org) series, indexed by DBLP, Google Scholar, Scopus and others.

All the accepted papers will be given a presentation slot during EVIA.

EVIA is open not only to all attendees at NTCIR but it also welcomes submissions of people who are not NTCIR participant but wish to discuss the future of information access evaluation in a stimulating, friendly and welcoming environment.



We invite submissions of regular papers (up to 8-10 pages) and short papers (up to 4 pages). Submissions must be in English, in PDF format, and must use standard ACM SIGIR templates, available at ACM SIGIR templates for both LaTeX and Word.

Papers must report work that is not previously published, not accepted for publication elsewhere, and not currently under review elsewhere.

Submissions will be subject to double-blind reviewing and should not contain any author identification.

Papers should be submitted electronically conference submission system at the conference submission system.



EVIA 2019 introduces a new two-steps submission process where authors can target either an early submission deadline or a late submission deadline. The idea is to give authors who have already consolidated works and/or need an earlier notification for making travel arrangements the possibility of anticipating the typical review cycle for EVIA; on the other hand, authors who do not have strict travel constraints or are still developing their own new and groundbreaking ideas can benefit from more time and a deadline closer to the event itself.

It is intended that submissions which are rejected at the first round cannot be re-submitted at the second round.

*Deadline time is 11:59 p.m. (anywhere in the world).*
February 28, 2019
First round submission deadline*
March 29, 2019
First round notifications to authors*
April 29, 2019
Second round submission deadline* (extended)
May 16, 2019
Second round notifications to authors*
May 27, 2019
Camera ready (first and second submissions) due*
June 10, 2019
EVIA 2019 @NII, Tokyo, Japan
June 11-13, 2019
NTCIR-14 @NII, Tokyo, Japan

As you can note, the new two-steps submission process make the EVIA schedule quite tight. Therefore the will be NO submission deadline extension, also because the two different deadlines remove this need.



  • Nicola FerroUniversity of Padua, Italy (Co-Chair)

  • Ian SoboroffNational Institute of Standards and Technology (NIST), USA (Co-Chair)

  • Min ZhangTsinghua University, China (Co-Chair)

  • Nicholas BelkinRutgers University, USA

  • Charles ClarkeUniversity of Waterloo, Canada

  • Atsushi FujiiTokyo Institute of Technology, Japan

  • Donna HarmanNational Institute of Standards and Technology (NIST), USA

  • Gareth JonesDublin City University, Ireland

  • Diane KellyUniversity of Tennensee, USA

  • Aldo LipaniVienna University of Technology, Austria

  • Claudio LuccheseCa' Foscari University of Venice and ISTI CNR Pisa, Italy

  • Maria MaistroUniversity of Padua, Italy

  • Stefano MizzaroUniversity of Udine, Italy

  • Alistair MoffatUniversity of Melbourne, Australia

  • Tetsuya SakaiWaseda University, Japan

  • Mark SandersonRMIT University, Australia

  • Ruihua SongMicrosoft Research Asia, China

  • Paul ThomasMicrosoft, Australia

  • Justin ZobelUniversity of Melbourne, Australia



Last Modified: 2019-06-10