[Date Prev][Date Next][Date Index]

[ntcir:222] [Fwd: [ntcir:221] Call for NTCIR-7 Task Proposals]




# This message contains a Japanese passage at the beginning.
# The rest is in English.

酒井と申します。NII神門先生の代理でメイルを書いております。

まだNTCIR-6の会議も終わっていませんが、そろそろNTCIR-7以降、
どのような情報アクセスタスクをやるかを考える必要があります。

下記の要領にしたがってタスクをご提案いただき、
それが採択されれば、どなたでもタスク主催者になれます。
是非ご検討ください。ご不明な点は、kando at nii.ac.jp
までお問い合わせください。
生煮えの段階でご相談いただいても結構です。

なお、TRECと同様、既存タスク・新規タスクいずれの場合も一応
提案書(proposal form)の作成をお願いしております。

どうぞよろしくお願いいたします。

-------- Original Message --------
Subject: [ntcir:221] Call for NTCIR-7 Task Proposals
Date: Fri, 01 Dec 2006 10:42:09 +0900
From: Tetsuya Sakai <tetsuya.sakai@xxxxxxxxxxxxx>
To: ntcir@xxxxxxxxx

Hi all,

NTCIR-6 isn't over yet, but we need to start thinking about
what to do for NTCIR-7 (Workshop Meeting: Dec 2008).

Attached are draft Call for NTCIR-7 Task Proposals and
a Proposal Form. If you are interested in coordinating a task (both
existing and new), please have a look.

Questions and comments are welcome.

- Tetsuya Sakai

############################################################################

	Procedure for Proposing NTCIR Tasks
						Nov 23, 2006 version 1.1
						NTCIR Program Committee

We welcome task proposals that are relevant to Information Access
research and are for noncommercial purposes. Accepted task
coordinators may possibly receive funding and/or assistance from
NII for obtaining the necessary data and for carrying out the tasks.
NII may also be able to provide the coordinators with NTCIR's legacy,
such as document collections from the past NTCIR tasks. NTCIR has
attracted participants from over fifteen countries. By coordinating
a task at NTCIR, you can create a new research area and establish
an international community!


SUBMITTING YOUR TASK PROPOSAL:

Please submit your task proposal in writing using the NTCIR-7 TASK
PROPOSAL FORM, even if you are a coordinator of an existing task.
For NTCIR-7, the proposals are due by *February 28, 2007*.
However, if you plan to use any new document collections that are
NOT yet part of NTCIR data, please contact the NTCIR Program
Committee (PC) Chair, Noriko Kando by *December 31, 2006*
even before completing the proposal form.
This is because it often takes time to obtain the necessary rights to
use new document data.

The proposals will be assessed by the NTCIR PC based on several criteria
described below and the accepted tasks will be announced shortly after
the NTCIR-6 meeting (May 15-18, 2007).
The task proposal document should be sent as an email attachment to
kando at nii.ac.jp with the subject: NTCIR-7 task proposal.

For an overview of the past and existing NTCIR Tasks as well as some
possibilities for future research at NTCIR, please refer to the NTCIR
TECHNOLOGY MAP (to be released in Dec 2006).


PROPOSAL ASSESSMENT CRITERIA:

The task proposals will be assessed from several viewpoints including:
- Relevance to NTCIR's scope
- Importance/usefulness to the research community and to the users
- Novelty
- Task feasibility, including expected number of participants and
    data availability
- Appropriateness of evaluation methods
- Fit with other NTCIR tasks and other non-NTCIR evaluation efforts
    such as TREC and CLEF
- Total number of tasks


TASK RENEWAL POLICY:

NTCIR welcomes "active" tasks, in which coordinators and participants
interact and collaborate with one another to successfully achieve the
task goals.

We would like every coordinator and participant to really think about
their task and keep reviving it over the NTCIR rounds, for example,
by redefining the task specifications in order to meet more practical
needs or to pursue more focussed academic questions.

We therefore expect that:

- The task coordinators and participants will have round-table
    and/or net-based meetings in order to refine the task details.
    Thus, the task proposal document is only a starting point, and
    it should be revised as the coordinators and participants
    approach the task goal.

- Any task coordinator will step down after two consecutive
    NTCIR rounds, and will be replaced by another if the task
    is to be continued.
    This is for encouraging others to experience the fun (and
    the difficulties!) of coordinating a task and for allowing
    different ways to define and analyse the same task.
    A coordinator who has just stepped down can, however,
    propose a new task immediately.


IMPORTANT DATES:

December 31, 2006	Contact deadline for prospective task
                         coordinators who plan to use NEW document
                         collections
February 28, 2007	TASK PROPOSALS DUE
May 15-18, 2007		Task planning meetings for NTCIR-7 at
			the NTCIR-6 Workshop Meeting (NII, Tokyo)
May 2007		TASK ACCEPTANCE NOTIFICATION
June 2007		TASK COORDINATORS MEET-UP
July-August 2007	NTCIR-7 Call for Participation
Spring-Summer 2008	NTCIR-7 formal runs and evaluation
October 2008		NTCIR-7 papers due
December 2008		NTCIR-7 Workshop meeting (NII, Tokyo)

We look forward to receiving your proposals!

############################################################################

	NTCIR-7 Task Proposal Form
						Nov 23, 2006 version 1.0

	TASK TITLE AND ABBREVIATED NAME:
# e.g. "User Satisfaction Task"

	ABBREVIATED TASK NAME (PRONUNCIATION):
# e.g. "USAT (you-sat)"

	CONTACT DETAILS OF TASK COORDINATORS:
# Names, affiliations and email addresses.
# List up at least TWO coordinators.

	TASK DEFINITION:
# What is the system required to do, and for what kind of users, and for
what purpose?
# What kind of input data are provided to the system, and in what format?
# What kind of output is the system expected to return, and in what format?
# Any subtasks?

	BACKGROUND AND OBJECTIVES:
# What is your motivation for doing this task?
# Why is it important to start the task NOW?
# What is the novelty of the task? How is it different from other tasks?
# What do you expect to learn from the task results?
# What will be the impact of the results? To whom will they be useful?
# Provide bibliographic references that serve as the background of your
proposal.

	DATA TO BE CONSTRUCTED/USED:
# How do you plan to obtain the data, e.g., document collections and
search requests?
# Data size and data cost?
# How do you plan to prepare "right answers" (if applicable)?
# After completion of the task, what data will be available to the
participants/public?

	HUMAN RESOURCES:
# Expected amount of human effort (e.g., in man-months) for user
evaluation, manual preparation of "right answers" (e.g. relevance
assessments) etc.
# Expected amount of "right answers"
# Who is going to provide the human resources?
#  (a) You, the coordinator
#  (b) NII
#  (c) The coordinator + NII
# What will be the requirements of the people who are going to be the
involved in the user evaluation and/or providing the "right answers"?
# How will the coordinators from overseas collaborate with one other?

	BUDGET:
# Do you have your own budget? If so how much? Who are your sponsors?
# Or do you expect NII to cover X% of the expenses?
# For coordinators overseas: About how many trips to Japan will be
required? What will be the travel expenses?

	POSSIBLE EVALUATION CRITERIA/METRICS:
# Why are the criteria/metrics appropriate? Why not others? What are the
pros and cons? Provide bibliographic references.

	EXPECTED NUMBER OF PARTICIPANTS/PARTICIPATING COUNTRIES:

	SCHEDULE:
# What will happen between now and the NTCIR-7 Workshop Meeting Day,
# e.g. task coordinators meet-up; first call for participation;
round-table meetings; dry run/formal run deadlines; results release...
# TENTATIVE SCHEDULE FOR NTCIR-7: Oct 2008 all papers due; Dec 2008
NTCIR-7 meeting.

	ROADMAP:
# Do you plan to continue your task at NTCIR-8, NTCIR-9,...?
# What will be the new focus?
# Or will there be a related but new task at NTCIR-8?

	COMMENTS:
# Anything that you might want to say to the NTCIR Program Committee.

############################################################################