Frequentist performance of sequential testing based on predictive power

Question

Suppose we perform a hypothesis test from a random sample $(x_i)_{i=1}^n$, assuming for instance $x_i \sim_{\text{iid}} {\cal N}(\theta, 1)$ and $H_0=\{\theta=0\}$ for simplicity. If the test fails to reject $H_0$, it is natural to ask oneself "if I collect more data, what could happen?".

This question can be solved by assigning a "predictive distribution" to the new data, and then calculate the probability to reject $H_0$ for a new sample assumed to be distributed according to this predictive distribution. A naive way (suffering from a lack of probabilist interpretation) consists in saying that the new data are $\sim_{\text{iid}} {\cal N}(\hat\theta, 1)$ where $\hat\theta$ is an estimate of $\theta$ obtained from the actual sample $(x_i)_{i=1}^n$. In the Bayesian framework there is a clear notion of predictive distribution.

Assuming we consider either the "naive" predictive distribution or the Bayesian predictive distribution derived from a noninformative prior, how to highlight/assess the frequentist performance of the predictive power ?

For instance, assume that we adopt the following sequential testing methodology:

• Collect a sample $(x_i)_{i=1}^n$ with $n=10$
• If $H_0$ is rejected then stop.
• Otherwise use the predictive distribution to evaluate the required new sample size to get $80\%$ predictive power to reject $H_0$.
• Then collect the new sample and perform the test.

Adopting this methodology, what about the probability to reject $H_0$ in function of $\theta$ ?

It would be easy to explore such questions using simulations, but does there exist some theoretical results about such questions ?

Answer 1

There is a developing theory of adaptive design that can be approached in either the frequentist or the Bayesian framework. For the frequentist approach with a small chapter on the Bayesian approach see this text by Chow and Chang. A thorough treatment of the bayesian approach can be found in Berry, Carlin, Lee and Muller.

Answer 2

Here is an example of using the Bayesian predictive distribution for planning a new experience: Sample size determination for a Gaussian mean It is not really related to the power of a hypothesis test but the approach is of the same spirit as the question of the OP: the goal is to guarantee a given probability of success for a certain event, similarly to the question of the OP in the context of hypothesis testing.

The conclusion is that indeed the Bayesian prediction approach (with an appropriate noninformative prior) controls the frequentist probability of success. Here the observation of interest is a sample standard deviation, and the Bayesian predictive distribution of this standard deviation enjoys the "frequentist-matching property": Bayesian $100p\%$-prediction intervals coincidentally are frequentist $100p\%$-prediction intervals.

In the context of the OP, the same result should occur if this predictive "frequentist-matching property" occurs when the observation of interest is the test statistic.