# Frequentist performance of sequential testing based on predictive power

Suppose we perform a hypothesis test from a random sample $(x_i)_{i=1}^n$, assuming for instance $x_i \sim_{\text{iid}} {\cal N}(\theta, 1)$ and $H_0=\{\theta=0\}$ for simplicity. If the test fails to reject $H_0$, it is natural to ask oneself "if I collect more data, what could happen?".

This question can be solved by assigning a "predictive distribution" to the new data, and then calculate the probability to reject $H_0$ for a new sample assumed to be distributed according to this predictive distribution. A naive way (suffering from a lack of probabilist interpretation) consists in saying that the new data are $\sim_{\text{iid}} {\cal N}(\hat\theta, 1)$ where $\hat\theta$ is an estimate of $\theta$ obtained from the actual sample $(x_i)_{i=1}^n$. In the Bayesian framework there is a clear notion of predictive distribution.

Assuming we consider either the "naive" predictive distribution or the Bayesian predictive distribution derived from a noninformative prior, how to highlight/assess the frequentist performance of the predictive power ?

For instance, assume that we adopt the following sequential testing methodology:

• Collect a sample $(x_i)_{i=1}^n$ with $n=10$
• If $H_0$ is rejected then stop.
• Otherwise use the predictive distribution to evaluate the required new sample size to get $80\%$ predictive power to reject $H_0$.
• Then collect the new sample and perform the test.

Adopting this methodology, what about the probability to reject $H_0$ in function of $\theta$ ?

It would be easy to explore such questions using simulations, but does there exist some theoretical results about such questions ?

-

There is a developing theory of adaptive design that can be approached in either the frequentist or the Bayesian framework. For the frequentist approach with a small chapter on the Bayesian approach see this text by Chow and Chang. A thorough treatment of the bayesian approach can be found in Berry, Carlin, Lee and Muller.

-
(+1) Dumb question from an interested non statistician: What about this book (Group Sequential Methods with Applications to Clinical Trials) ? How does it compare to the others ? –  steffen Sep 12 '12 at 10:36
@steffen The book you cite by Jennison and Turnbull is perhpas the best book available on groups equential methods. Adaptive trials are a little different though. Traditional group sequential methods use sequential stopping rules by taking groups of samples. The rules are setup in advance and the data is used solely for the purpose of deciding when to stop and what conclusion to reach whne you stop. Adaptive methods allow you to make specified changes to the design at the interim points. –  Michael Chernick Sep 12 '12 at 10:52
Examples are (1) changes the number of treatment groups, (2) adjusting the number of samples to take for the next stage. Hence the term adapt meaning that adaptations to the design are made at interim looks. –  Michael Chernick Sep 12 '12 at 10:54
I see, thank you very much ! –  steffen Sep 13 '12 at 6:44
The conclusion is that indeed the Bayesian prediction approach (with an appropriate noninformative prior) controls the frequentist probability of success. Here the observation of interest is a sample standard deviation, and the Bayesian predictive distribution of this standard deviation enjoys the "frequentist-matching property": Bayesian $100p\%$-prediction intervals coincidentally are frequentist $100p\%$-prediction intervals.