We present two new measures of retrieval effectiveness, inspired by Graded Average Precision (GAP), which extends Average Precision (AP) to graded relevance judgements. Starting from the random choice of a user, we define Extended Graded Average Precision (xGAP) and Expected Graded Average Precision (eGAP), which are more accurate than GAP in the case of a small number of highly relevant documents with high probability to be considered relevant by the users. The proposed measures are then evaluated on TREC 10, TREC 14, and TREC 21 collections showing that they actually grasp a different angle from GAP and that they are robust when it comes to incomplete judgments and shallow pools.

Rethinking How to Extend Average Precision to Graded Relevance

FERRANTE, MARCO;FERRO, NICOLA;MAISTRO, MARIA
2014

Abstract

We present two new measures of retrieval effectiveness, inspired by Graded Average Precision (GAP), which extends Average Precision (AP) to graded relevance judgements. Starting from the random choice of a user, we define Extended Graded Average Precision (xGAP) and Expected Graded Average Precision (eGAP), which are more accurate than GAP in the case of a small number of highly relevant documents with high probability to be considered relevant by the users. The proposed measures are then evaluated on TREC 10, TREC 14, and TREC 21 collections showing that they actually grasp a different angle from GAP and that they are robust when it comes to incomplete judgments and shallow pools.
2014
Information Access Evaluation -- Multilinguality, Multimodality, and Interaction. Proceedings of the Fifth International Conference of the CLEF Initiative (CLEF 2014)
File in questo prodotto:
Non ci sono file associati a questo prodotto.
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11577/3147576
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 2
  • ???jsp.display-item.citation.isi??? ND
social impact