A Flexible Defense Against the Winner’s Curse

Abstract:
Across science and policy, decision-makers often need to draw conclusions about the best candidate among competing alternatives. For instance, researchers may seek to infer the effectiveness of the most successful treatment or determine which demographic group benefits most from a specific treatment. Similarly, in machine learning, practitioners are often interested in the population performance of the model that empirically performs best. However, cherry-picking the best candidate leads to the winner’s curse: the observed performance for the winner is biased upwards, rendering conclusions based on standard measures of uncertainty invalid. We introduce a novel approach for valid inference on the winner. Our method is flexible: it handles arbitrary dependence between candidates and is entirely nonparametric. It automatically adapts to the level of selection bias; in particular, it recovers standard, uncorrected inference when the winner stands out and becomes increasingly conservative when there are multiple competitive candidates. The robust underpinnings of the method allow easy extensions to important related problems, such as inference on the top k winners, inference on the value and identity of the population winner, and inference on “near-winners.”