[Date Prev][Date Next][Date Index]

[ntcir:228] Call for NTCIR-7 Task Proposals:Deadline extended April 15

Dear all,

As several groups asked, the deadline of the NTCIR-7
task proposal has been extended to APRIL 15, 2007.

Please understand that no NTIR-6 tasks are guaranteed
the continuation to NTCIR-7 without approved proposals.
A "new blood" to refresh NTCIR and tasks is welcome as
well as veterans.

Evaluation must evlove according to the technology
improvement and the social changes. What kind of tasks
are needed to encourage such technological improvement
so far and in near future?  What kind of document genres
and user groups are critical in information access
technology research for now and near future?

Web is definitely a big issue  -- but Web is no more one
single document genre but qutie heterogeous complex and
its usage and usar groups are heterogeous as well.

How devices (e.g. mobile phone)  affected the information
access technologies?

On the other hand, there are also many traditional document
genres or user groups need sophisticated information
access technologies.  In global enterprise, personal
desk-top, scientific, legal, intellectual property, etc.

How to tackle such diversity?  How to tackle the size
of the collection? What kind of evaluation infrastrucure is
needed to enhance the research in information access
and to speed the research towards real world.

Your input and contribution are always welcome.

Noriko.

############################################################################

	Procedure for Proposing NTCIR Tasks
						Nov 23, 2006 version 1.1
                                                Feb 28, 2007 version 1.2
						NTCIR Program Committee

We welcome task proposals that are relevant to Information Access
research and are for noncommercial purposes. Accepted task
coordinators may possibly receive funding and/or assistance from
NII for obtaining the necessary data and for carrying out the tasks.
NII may also be able to provide the coordinators with NTCIR's legacy,
such as document collections from the past NTCIR tasks. NTCIR has
attracted participants from over fifteen countries. By coordinating
a task at NTCIR, you can create a new research area and establish
an international community!


SUBMITTING YOUR TASK PROPOSAL:

Please submit your task proposal in writing using the NTCIR-7 TASK
PROPOSAL FORM, even if you are a coordinator of an existing task.
For NTCIR-7, the proposals are due by *February 28, 2007*.
However, if you plan to use any new document collections that are
NOT yet part of NTCIR data, please contact the NTCIR Program
Committee (PC) Chair, Noriko Kando by *December 31, 2006*
even before completing the proposal form.
This is because it often takes time to obtain the necessary rights to
use new document data.

The proposals will be assessed by the NTCIR PC based on several criteria
described below and the accepted tasks will be announced shortly after
the NTCIR-6 meeting (May 15-18, 2007).
The task proposal document should be sent as an email attachment to
kando at nii.ac.jp with the subject: NTCIR-7 task proposal.



PROPOSAL ASSESSMENT CRITERIA:

The task proposals will be assessed from several viewpoints including:
- Relevance to NTCIR's scope
- Importance/usefulness to the research community and to the users
- Novelty
- Task feasibility, including expected number of participants and
    data availability
- Appropriateness of evaluation methods
- Fit with other NTCIR tasks and other non-NTCIR evaluation efforts
    such as TREC and CLEF
- Total number of tasks


TASK RENEWAL POLICY:

NTCIR welcomes "active" tasks, in which coordinators and participants
interact and collaborate with one another to successfully achieve the
task goals.

We would like every coordinator and participant to really think about
their task and keep reviving it over the NTCIR rounds, for example,
by redefining the task specifications in order to meet more practical
needs or to pursue more focussed academic questions.

We therefore expect that:

- The task coordinators and participants will have round-table
    and/or net-based meetings in order to refine the task details.
    Thus, the task proposal document is only a starting point, and
    it should be revised as the coordinators and participants
    approach the task goal.

- Any task coordinator will step down after two consecutive
    NTCIR rounds, and will be replaced by another if the task
    is to be continued.
    This is for encouraging others to experience the fun (and
    the difficulties!) of coordinating a task and for allowing
    different ways to define and analyse the same task.
    A coordinator who has just stepped down can, however,
    propose a new task immediately.


IMPORTANT DATES:

December 31, 2006	Contact deadline for prospective task
                         coordinators who plan to use NEW document
                         collections  ** as soon as possible **

April 15, 2007  	TASK PROPOSALS DUE  *** Extended! ***

May 15-18, 2007		NTCIR-6 Workshop Meeting (NII, Tokyo)
                        (Discussion for Task Planning and PC meeting)

May 2007		TASK ACCEPTANCE NOTIFICATION
June 2007		TASK COORDINATORS MEET-UP
July-August 2007	NTCIR-7 Call for Participation
Spring-Summer 2008	NTCIR-7 formal runs and evaluation
October 2008		NTCIR-7 papers due
December 2008		NTCIR-7 Workshop meeting (NII, Tokyo)

We look forward to receiving your proposals!

############################################################################

	NTCIR-7 Task Proposal Form
						Nov 23, 2006 version 1.0

	TASK TITLE AND ABBREVIATED NAME:
# e.g. "User Satisfaction Task"

	ABBREVIATED TASK NAME (PRONUNCIATION):
# e.g. "USAT (you-sat)"

	CONTACT DETAILS OF TASK COORDINATORS:
# Names, affiliations and email addresses.
# List up at least TWO coordinators.

	TASK DEFINITION:
# What is the system required to do, and for what kind of users, and for
what purpose?
# What kind of input data are provided to the system, and in what format?
# What kind of output is the system expected to return, and in what format?
# Any subtasks?

	BACKGROUND AND OBJECTIVES:
# What is your motivation for doing this task?
# Why is it important to start the task NOW?
# What is the novelty of the task? How is it different from other tasks?
# What do you expect to learn from the task results?
# What will be the impact of the results? To whom will they be useful?
# Provide bibliographic references that serve as the background of your
proposal.

	DATA TO BE CONSTRUCTED/USED:
# How do you plan to obtain the data, e.g., document collections and
search requests?
# Data size and data cost?
# How do you plan to prepare "right answers" (if applicable)?
# After completion of the task, what data will be available to the
participants/public?

	HUMAN RESOURCES:
# Expected amount of human effort (e.g., in man-months) for user
evaluation, manual preparation of "right answers" (e.g. relevance
assessments) etc.
# Expected amount of "right answers"
# Who is going to provide the human resources?
#  (a) You, the coordinator
#  (b) NII
#  (c) The coordinator + NII
# What will be the requirements of the people who are going to be the
involved in the user evaluation and/or providing the "right answers"?
# How will the coordinators from overseas collaborate with one other?

	BUDGET:
# Do you have your own budget? If so how much? Who are your sponsors?
# Or do you expect NII to cover X% of the expenses?
# For coordinators overseas: About how many trips to Japan will be
required? What will be the travel expenses?

	POSSIBLE EVALUATION CRITERIA/METRICS:
# Why are the criteria/metrics appropriate? Why not others? What are the
pros and cons? Provide bibliographic references.

	EXPECTED NUMBER OF PARTICIPANTS/PARTICIPATING COUNTRIES:

	SCHEDULE:
# What will happen between now and the NTCIR-7 Workshop Meeting Day,
# e.g. task coordinators meet-up; first call for participation;
round-table meetings; dry run/formal run deadlines; results release...
# TENTATIVE SCHEDULE FOR NTCIR-7: Oct 2008 all papers due; Dec 2008
NTCIR-7 meeting.

	ROADMAP:
# Do you plan to continue your task at NTCIR-8, NTCIR-9,...?
# What will be the new focus?
# Or will there be a related but new task at NTCIR-8?

	COMMENTS:
# Anything that you might want to say to the NTCIR Program Committee.

############################################################################