Evaluation Methodologies in Information Retrieval (Dagstuhl Seminar 13441)

This paper documents the program and the outcome of Dagstuhl Seminar 13441 “Evaluation Methodologies in Information Retrieval”, which brought together 42 participants from 11 countries. The seminar was motivated by the fact that today’s information retrieval (IR) applications can hardly be evaluated based on the classic test collection paradigm, thus there is a need for new evaluation approaches. The event started with five introductory talks on evaluation frameworks, user modeling for evaluation, evaluation criteria, measures, evaluation methodology, and new trends in IR evaluation. The seminar participants then formed working groups addressing specific aspects of IR evaluation, such as reliability and validity, task-based IR, learning as search outcome, searching for fun, IR and social media, graph search, domain-specific IR, interaction measures and models, and searcher-aware information access systems.