# Frequentist performance of sequential testing based on predictive power

Suppose we perform a hypothesis test from a random sample $(x_i)_{i=1}^n$, assuming for instance $x_i \sim_{\text{iid}} {\cal N}(\theta, 1)$ and $H_0=\{\theta=0\}$ for simplicity. If the test fails to reject $H_0$, it is natural to ask oneself "if I collect more data, what could happen?".

This question can be solved by assigning a "predictive distribution" to the new data, and then calculate the probability to reject $H_0$ for a new sample assumed to be distributed according to this predictive distribution. A naive way (suffering from a lack of probabilist interpretation) consists in saying that the new data are $\sim_{\text{iid}} {\cal N}(\hat\theta, 1)$ where $\hat\theta$ is an estimate of $\theta$ obtained from the actual sample $(x_i)_{i=1}^n$. In the Bayesian framework there is a clear notion of predictive distribution.

Assuming we consider either the "naive" predictive distribution or the Bayesian predictive distribution derived from a noninformative prior, how to highlight/assess the frequentist performance of the predictive power ?

For instance, assume that we adopt the following sequential testing methodology:

• Collect a sample $(x_i)_{i=1}^n$ with $n=10$
• If $H_0$ is rejected then stop.
• Otherwise use the predictive distribution to evaluate the required new sample size to get $80\%$ predictive power to reject $H_0$.
• Then collect the new sample and perform the test.

Adopting this methodology, what about the probability to reject $H_0$ in function of $\theta$ ?

It would be easy to explore such questions using simulations, but does there exist some theoretical results about such questions ?

The conclusion is that indeed the Bayesian prediction approach (with an appropriate noninformative prior) controls the frequentist probability of success. Here the observation of interest is a sample standard deviation, and the Bayesian predictive distribution of this standard deviation enjoys the "frequentist-matching property": Bayesian $100p\%$-prediction intervals coincidentally are frequentist $100p\%$-prediction intervals.