0001 C gakkai-0000010176 0 0001 A gakkai-0000010187 1 0001 C gakkai-0000010218 0 0001 C gakkai-0000010219 0 0001 C gakkai-0000010220 0 0002 A gakkai-0000010187 1 0002 A gakkai-0000010221 1 0002 C gakkai-0000010222 0 0002 C gakkai-0000010231 0
Relevance assessment is done in three grades, i.e., "relevant (A)", "partially relevant (B)", and "non-relevant". The documents which human assessors assessed as "non-relevant" have lable of "C". The coduments whose IDs were not included in the relevance assessment file were not assessed by human assessors but we supporsed that they are be "non-relevant" since they were not retrieved as candidates for relevant documents neverthless we have tried various search strategies.
Relevance assessment should be done in two grades ("0" and "1") in order to
calculate the recall and precision.
Therefore there are two types of relevance assessment files: "Relevant qrel" and "Patial Relevant qrel".
In the "Relevant qrel" file, the documents with A-judgement (relevant) are rated as
"Relevant (1)", and documents with B- or C-judgement are treated as "Non-Relevant (0)".
In the "Partial Relevant qrel" file, the documents with A- or B-judgement are rated as
"Relevant (1)", and documents with C-judgements are treated as "Non-Relevant".
The format of a relevance assessment file is as follows:
topic-ID dummy document-ID relevance-assesment ( "0" or "1")