gakkai-e-0000441590
NTCIR-1 : NACSIS Test Collection for Information Retrieval systems -1 (1) -Analysis of the Pooling and the Relevance Assessments-
Kuriyama,Kazuko / Eguchi,Koji / Nozue,Toshihiko / Kando,Noriko
National Convention
1999. 09. 28 - 1999. 09. 30
The purposes of this study are ; (1) to verify the effectiveness of the pooling method to construct a large scale test collection, (2) to examine whether the number of documents in a pool affects the evaluation of retrieval results, and (3) to verify the reliability of a test collection through investigating the effect of variations in relevance assessments have on the evaluation of search effectiveness since individual relevance assessments are known to be vary widely.For (1) and (2), we carried out an experiment using the qrel's (queries and their relevance assessments) and the submitted search results for the test of the NTCIR-1 workshop. The result of it is that we verified the efficiency and effectiveness of the pooling, the exhaustiveness of the relevance assessments, the reliability of the evaluation using the test collection based on the pooling method, and the effectiveness of the modified Move-to-Front pooling method. For (3) we compared search effectiveness of retrieval strategies using the three different sets of relevance assessments judged by the assessor A, B and the final judgment based on the negotiation between A and B.As a result, we found very high similarity among the rankings of retrieval systems produced using different set of relevance assessments when a sufficient number of search topics were used in the evaluation.
Test Collection // Pooling // Information Retrieval // NACSIS Test Collection for Information Retrieval systems // MTF heuristic method // exhanstiveness
Information Processing Society of Japan