Sequential testing of multiple probabilities (binomial variables)

Question

Task:
I have e.g. 1 control group and 7 experimental groups which are treated differently. All groups have a dependent binomial variable: 1 = success, 0 = no success.

I want to conduct sequential tests, which version is "performing" better, between the control group and the experimental groups and stop the tests, when one of the experimental group is better that the control group (or it is proven, that the Null hypothesis is true).

Problem:
So in my opinion two problems arise: alpha I error inflation by sequential testing and alpha I error inflation by testing multiple hypothesis (per timepoint)

My research so far:
I found group sequential methods, Alpha-Spending approaches and SPRT to solve the sequential testing problem. And I read the sequential updating approach of Bayesian Methods in the book of Kruschke.

For the multiple hypothesis testing (per time), there are some techniques which control the overall alpha error rate (per time), like Bonferroni, Hochberg's step-up procedure and others.

But I haven't found how to combine the methods. Nevertheless, group sequential methods as well as Alpha-Spending approaches seem to be a bit inflexible. SPRT und Bayesian Updating seem to fit best for the need to have a small sample number, being able to test after every participant and not to have a predefined endpoint of the study.

Bayesian updating (with ROPE), as far a I know, does not have something like a predefinable alpha error rate (and no correction for multiple variables?). SPRT also does not support multiple variables. Please correct me if I am wrong. Do you have any suggestions where to dig deeper?

Best Regards Andreas

Answer 1

The reason you don't see notions like a "predefinable alpha error rate" with Bayesian updating is that the update of the posterior often accounts for these effects. If the sole decision that's needed is when to stop, then that can be posed as a Bayesian hierarchical model. If tests need to be done along the way, then, yes, life is more complicated and you'll need to study, 'cause there's no quick answer.

First paper to check out is Gelman, Hill, Yajima, "Why We (Usually) Don't Have To Worry About Multiple Comparisons", http://www.stat.columbia.edu/~gelman/research/published/multiple2f.pdf

The second, more important reference is Section 7.6 ("Sequential analysis") and the related subsection 8.2.4 of Carlin and Louis, BAYESIAN METHODS FOR DATA ANALYSIS (3rd edition), 2009, pages 346ff and 398ff.

All that said, while I can't delve any more deeply into your problem of groups and their success or failure in your model, I suspect you may be thresholding some measure of performance to obtain success or failure. If so, information may be being thrown away unnecessarily and, so, whatever you determine is a stopping point may be more conservative than necessary ... meaning later. If so, the measures will need a different model than binomial, but you'll gain. For more, see the talk by Dr Frank Harrell, Jr, 2010, and its references, Information Allergy.