Impact of query sample selection bias on information retrieval system ranking

Information Retrieval (IR) effectiveness measures commonly assume that the experimental query sets consist of randomly drawn queries that represent the population of queries submitted to IR systems. In many practical situations, however, this assumption is violated, in a problem known as sample selection bias. It follows that the systems participating in evaluation campaigns are ranked by biased estimators of effectiveness. In this paper, we address the problem of query sample selection bias in machine learning terms and study experimentally how retrieval system rankings are affected by it. To this end, we apply a number of retrieval effectiveness measures and query probability estimation methods useful to correct sample selection bias. We report that the ranking of the most effective systems and that of the least effective systems is fairly affected by query sample selection bias, while the ranking of the average systems is much more affected. We also report that the measure of bias d...