<REC>
<ACCN>gakkai-e-0000441590</ACCN>
<TITE TYPE="alpha">NTCIR-1 : NACSIS Test Collection for Information Retrieval systems -1 (1) -Analysis of the Pooling and the Relevance Assessments-</TITE>
<AUPE TYPE="alpha">Kuriyama,Kazuko / Eguchi,Koji / Nozue,Toshihiko / Kando,Noriko</AUPE>
<CNFE>National Convention</CNFE>
<CNFD>1999. 09. 28 - 1999. 09. 30</CNFD>
<ABSE TYPE="alpha"><ABSE.P>The purposes of this study are ; (1) to verify the effectiveness of the pooling method to construct a large scale test collection, (2) to examine whether the number of documents in a pool affects the evaluation of retrieval results, and (3) to verify the reliability of a test collection through investigating the effect of variations in relevance assessments have on the evaluation of search effectiveness since individual relevance assessments are known to be vary widely.</ABSE.P><ABSE.P>For (1) and (2), we carried out an experiment using the qrel's (queries and their relevance assessments) and the submitted search results for the test of the NTCIR-1 workshop. The result of it is that we verified the efficiency and effectiveness of the pooling, the exhaustiveness of the relevance assessments, the reliability of the evaluation using the test collection based on the pooling method, and the effectiveness of the modified Move-to-Front pooling method. For (3) we compared search effectiveness of retrieval strategies using the three different sets of relevance assessments judged by the assessor A, B and the final judgment based on the negotiation between A and B.As a result, we found very high similarity among the rankings of retrieval systems produced using different set of relevance assessments when a sufficient number of search topics were used in the evaluation.</ABSE.P></ABSE>
<KYWE TYPE="alpha">Test Collection // Pooling // Information Retrieval // NACSIS Test Collection for Information Retrieval systems // MTF heuristic method // exhanstiveness</KYWE>
<SOCE TYPE="alpha">Information Processing Society of Japan</SOCE>
</REC>