Query Term Expansion Task

Objectives of  the Task

Query Term Expansion Task aims to discuss several issues that are related to the gap between query terms selected by a user and his/her information need. Most of the people only use few words to represent their information need in web information retrieval. Since these query words are not precise and well-informed enough to select relevant documents, user may select documents that do not contains a part of query words as relevant ones.

Therefore, query term expansion is necessary to find out relevant documents exhaustively. In this pilot task, we would like to discuss following issues.

• A method to find query terms that are useful to find relevant documents.
• A method to evaluate effectiveness of query terms.

Issues to be discussed for the task design and evaluation

We don't have concrete task definition and evaluation criteria at this moment. At first we would like to enumerate issues for the task design and evaluation.

Issues for task design

User (assessor) feedback

1. pre-defined relevance judgment (limited number)
   Use relevant documents information that are generated by preliminary search
   
2. interactive relevance judgment
   Each participants asks assessor to judge relevancy of top-ranked documents
3. pseudo-relevance judgment
   Use pseudo-relevant documents
   

Information used for query expansion

• Relevant documents(pre-defined or interactive)
  Use relevant document information judged by a user (assessor)
• Related category (pre-defined(?) or interactive)
  Information that are used for selecting documents and/or keywords cluster
• User information

Issues for evaluation

Types of the retrieval task

In order to evaluate effectiveness of query terms from the view point of recall-oriented, it is better to use survey type retrieval task.

User oriented evaluation

Evaluate expanded query terms by the assessors in following two method.

• We can ask the assessor to evaluate expanded query interactively.
• We ask the assessor to select useful terms from pre-defined relevant documents.
  

System dependent evaluation (Usage of reference IR system)

• Evaluation based on statistical information
  Evaluate expanded query terms by using statistical information (e.g., Usage of Robertson/Sparck-Jones formula by using all relevant documents information)

• Evaluation based on recall-oriented and precision-oriented
  Some terms are effective for improving precision and some ones for recall.
  

• Evaluation based on system performance
  Evaluate query terms by using performance measure of reference IR system.

Candidate Task Design

• Query term expansion for target retrieval task
  

• Automatic query term expansion
• Information used for query expansion: Initial query
  
• Submission results
  
• Ranked list of query term expansion candidates
  
• Ranked retrieval results by using initial query and expanded query
• Query term expansion for survey retrieval task
  
• Automatic query term expansion
• Information used for query expansion: Initial query
  
• Submission results
  
• Ranked list of query term expansion candidates
  
• Ranked retrieval results by using initial query and expanded query
• Use pre-defined relevant documents information
  
• Information used for query expansion: Initial query + 3-5 relevant documents
  
• Submission results
  
• Ranked list of query term expansion candidates
  
• Ranked retrieval results by using initial query and expanded query
• Interactive relevance judgment
  
• Information used for query expansion: Initial query
  
• Submission results
  
• Initial
• Questions to the assessor with top-ranked retrieval results
  
• Second
• Ranked list of query term expansion candidates
  
• Ranked retrieval results by using initial query and expanded query

Contact Information

Discussion and information exchange regarding the task:

We would like to use a mailing list (ntc-webj-qt) for discussing how to design and evaluate this task. First language of this mailing list is Japanese, but you may also use English for the mailing list. Everybody who is interested in can join, while only list members can submit messages. If you would like to join this mailing list, please refer to this web page (this mailing list is not listed there). If there are any difficulty, please contact the following WEB Task Administrators.

Technical questions and others:

WEB Task Organizers at ntcweb-org

Procedural and administrative matters:

WEB Task Administrators at ntcadm-web