WORKSHOP PROGRAM 13:00-18:00 (Mid-Conference Hall 2.3.4)

OPENING REMARKS 13:00-13:05 (Mark Sanderson and Tetsuya Sakai)

INVITED PAPER SESSION I - New Asian IR Initiatives 13:05-14:45 (Chair: Tetsuya Sakai)

13:05-13:25
Introduction of the HTRDP Chinese IR Evaluation
Le Sun (Chinese Academy of Sciences)

13:25-13:45
Vietnamese Text Retrieval: Test Collection and First Experimentations
Ho Bao Quoc (Vietnam National University)

13:45-14:05
Initiative for Indian Language IR Evaluation
Prasenjit Majumder, Mandar Mitra, Swapan Parui and Pushpak Bhattacharyya (Indian Statistical Institute)

14:05-14:25
Evaluation of Web Search Engines on Thai Queries
Virach Sornlertlamvanich (Thai Computational Linguistics Laboratory)

14:25-14:45
Questions and Discussions

COFFEE BREAK 14:45-15:00

INVITED PAPER SESSION II 15:00-15:35 (Chair: Mark Sanderson)

15:00-15:35
A Comparison of Pooled and Sampled Relevance Judgments in the TREC 2006 Terabyte Track
Ian Soboroff (NIST)

REGULAR PAPER SESSION 15:35-17:15 (Chair: Mark Sanderson)

15:35-16:00
On Penalising Late Arrival of Relevant Documents in Information Retrieval Evaluation with Graded Relevance
Tetsuya Sakai (NewsWatch, Inc.)

16:00-16:25
New Evaluation Measures for F-term Patent Classification
Yaoyong Li, Kalina Bontcheva and Hamish Cunningham (University of Sheffield)

16:25-16:50
WiQA: Evaluating Multi-lingual Focused Access to Wikipedia
Valentin Jijkoun and Maarten de Rijke (ISLA, University of Amsterdam)

16:50-17:15
A Proposal to Extend and Enrich the Scientific Data Curation of Evaluation Campaigns
Maristella Agosti, Giorgio Maria Di Nunzio and Nicola Ferro (University of Padua)

POSTER SESSION 17:15-18:00

Challenges to Evaluation of Multilingual Geographic Information Retrieval in GeoCLEF
Fredric Gey, Ray Larson, Mark Sanderson, Kerstin Bishoff, Thomas Mandl, Christa Womser-Hacker, Diana Santos, Paulo Rocha, Andres Montoyo, Giorgio Maria Di Nunzio and Nicola Ferro (University of California)

An Application of the NTCIR-WEB Raw-data Archive Dataset for User Experiments
Masao Takaku, Yuka Egusa, Hitomi Saito and Hitoshi Terai (Research Organization of Information and System)

User Satisfaction Task: A Proposal for NTCIR-7
Tetsuya Sakai (NewsWatch, Inc.)

---

PROGRAM COMMITTEE
Akiko Aizawa
James Allan
Javed Aslam
Nicholas Belkin
Paul Clough
Norbert Fuhr
Fred Gey
Donna Harman
David Harper
Djoerd Hiemstra
Gareth Jones
Noriko Kando
Jaana Kekalainen
Diane Kelly
Kui-Lam Kwok
Mounia Lalmas
Gary Geunbae Lee
Mun-Kew Leong
Douglas Oard
Yasushi Ogawa
Iadh Ounis
Carol Peters
Maarten de Rijke
Stephen Robertson
Ian Ruthven
Tetsuya Sakai
Mark Sanderson
Ian Soboroff
Andrew Turpin
Ross Wilkinson

---

CALL FOR POSTER SUBMISSIONS (MARCH 15, 2007 Extended to MARCH 31, 2007)

We are past the submission deadline for the REFEREED PAPERS but we have decided to additionally accept four-page POSTERS reporting late-breaking results on IR evaluation. Unlike the regular papers, the posters will NOT be reviewed. Anyone (NTCIR participants and non-participants) can submit a poster, but participants should make sure that the content of the poster is different from that of the NTCIR site report. We welcome poster submissions addressing one or more of the following topics:

• Test collections, their formation, evaluation metrics and evaluation environments
• Cross-conference comparison: NTCIR, CLEF, TREC, DUC and others
• Evaluation of specific information access tasks, e.g., for education and training
• Evaluation of query-specific approaches
• Evaluation of multilingual, multimedia and/or mobile information access
• Real-world search topics and/or evaluation in vivo
• Novel information access tasks and their evaluation

SUBMISSION GUIDELINES:
Please use latex8.sty and sample tex source file (ntcir_author.tex) available at the NTCIR-5 website (not NTCIR-6) but for the EVIA posters please note that the following special instructions apply:
- NUMBER OF PAGES:
We allow up to 4 pages for posters. Any submissions that are longer will be removed automatically from our submission list.
- PAPER SUBMISSION URL:
Submit your PDF file here. You can keep revising your submission at the above site until the deadline (March 31).
- DOUBLECHECK YOUR MANUSCRIPT!
Your poster will NOT be reviewed, so make sure there are no typos, etc.

---

CALL FOR PAPERS
EVIA 2007: The First International Workshop on Evaluating Information Access
May 15, 2007 @ National Institute of Informatics, Tokyo, Japan.

Workshop Co-Chairs:
Tetsuya Sakai (NewsWatch, Inc., Japan) and Mark Sanderson (University of Sheffield, UK)

Evaluation of Information Retrieval, Question Answering and Text Summarisation systems has been central to Information Access research for decades. As retrieval has become more pervasive and diverse, the need for effective and efficient evaluation has never been more important. Following the success of Open Submission Sessions over the last two years, NTCIR will hold a pre-meeting workshop called EVIA 2007, which will run the day before the main sessions of NTCIR start. We welcome submissions of high-quality papers from participants and non-participants, especially those addressing one or more of the following topics:

• Test collections, their formation, evaluation metrics and evaluation environments
• Cross-conference comparison: NTCIR, CLEF, TREC, DUC and others
• Evaluation of specific information access tasks, e.g., for education and training
• Evaluation of query-specific approaches
• Evaluation of multilingual, multimedia and/or mobile information access
• Real-world search topics and/or evaluation in vivo
• Novel information access tasks and their evaluation

The submitted papers will be blind-reviewed and those accepted will be presented orally at the Workshop on May 15, 2007. The accepted papers will be included in the online and printed versions of NTCIR-6 Proceedings. We expect the workshop to be more competitive than the previous Open Submission Sessions held at NTCIR-4 and 5.

IMPORTANT DATES:
January 5, 2007: Submissions due
February 15, 2007: Notifications of acceptance
March 15, 2007: Final manuscripts due
May 15, 2007: EVIA 2007 takes place

SUBMISSION GUIDELINES:
Please use latex8.sty and sample tex source file (ntcir_author.tex) available at the NTCIR-5 website (not NTCIR-6) but for EVIA 2007 please note that the following special instructions apply:
- NUMBER OF PAGES:
We allow up to 12 pages because some evaluation papers may involve lots of results and analyses.
- ANONYMIZE!
To facilitate blind reviewing, do NOT include the author names in your manuscript, and "anonymize" your own citations to the best of your ability.
- WHERE TO SUBMIT YOUR PAPER:
Submit your PDF file here. You can also revise your submission at the above site

Finally, please note that DUPLICATE SUBMISSIONS, if detected, will be rejected unconditionally. That is, submitting the same paper to EVIA 2007 and to another conference simultaneously is strictly prohibited.

If you have any questions, please contact:
Tetsuya Sakai or Mark Sanderson

---