sMARE: An enhanced query performance prediction evaluation approach

QPP has been studied extensively in the IR community over the last two decades. Nevertheless, the Query Performance Prediction (QPP) field still lacks sound theoretical evaluation methodologies. In this work∗, we re-examined the existing evaluation methodology commonly used for QPP, and propose a new approach. Our key idea is to model QPP performance as a distribution instead of relying on point estimates. Our work demonstrates important statistical implications and overcomes key limitations imposed by the currently used correlation-based point-estimate evaluation approaches. This, in turns, enables the use of ANalysis Of VAriance (ANOVA) models for comparative analyses, permitting deeper analyses on the QPP models performance, and allowing to measure interactions between multiple factors.