Safe Testing
Peter Grünwald, Rianne de Heide and Wouter Koolen
Safe Testing
Peter Grünwald, Rianne de Heide and Wouter Koolen
Journal of the Royal Statistical Society Series B: Statistical Methodology, Volume 86, Issue 5, November 2024, Pages 1091–1128. doi.org/10.1093/jrsssb/qkae011 arxiv proc
The discussion meeting took place on January 24, 2024 (video).
Authors’ Reply to the Discussion of ‘Safe Testing’
Peter Grünwald, Rianne de Heide and Wouter Koolen
Journal of the Royal Statistical Society Series B: Statistical Methodology, Volume 86, Issue 5, November 2024, Pages 1163–1171. doi:10.1093/jrsssb/qkae069 proc
What is this paper about?
This paper develops e-values as a general currency of statistical evidence for tests that remain valid under optional continuation. It also introduces growth-rate optimality as an analogue of power in this sequential setting and develops constructions for composite testing problems.
Summary
We develop the theory of hypothesis testing based on the e-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e-values are safe: they preserve Type-I error guarantees under such optional continuation. We define growth-rate optimality (GRO) as an analogue of power in an optional-continuation context, and show how to construct GRO e-variables for general testing problems with composite null and alternative, emphasizing models with nuisance parameters. GRO e-values take the form of Bayes factors with special priors. We illustrate the theory using several classic examples including a one-sample safe t-test and the 2 × 2 contingency table.
Related popular writing and media
Topics
E-values and anytime-valid inference · Foundations of statistics, probability and learning