Sequential analysis involves performing sequential interim analysis till results are significant or till a maximum number of interim analyses is reached.

Sequential analysis sounds appealing especially since it may result in trial needing much less number of subjects than a randomized trial where sample size is calculated in advance.

Are there any disadvantages of sequential analysis? Why it is not used more often? Thanks for your insight.

  • 2
    $\begingroup$ From a frequentist perspective, sequential analysis is limited to a pretty small class of problems, like simple univariate hypothesis tests. Beyond that, things get really hard, fast. Some further disadvantages are that there is no institutional momentum behind sequential analysis in most pockets of industry, and there are fears that sequential analyses could easily be misused. $\endgroup$ – JTH May 13 '19 at 19:35

From a frequentist perspective, there are some clear disadvantages of a sequential analyses. That is, if we are concerned with preserving type I errors, we need to recognize that we are doing multiple comparisons: if I do 3 analyses of the data, then I have three non-independent chances to make a type I error and need to adjust my inference as such. There's a variety of methods for accounting for this, but in short, for a fixed sample size and significance level, all of them end up reducing power compared to waiting until all the data comes in. So if you're looking at the power/subjects ratio, you can't beat a fixed analysis, although as you point out, often that's not necessarily the most important metric.

Theoretically, from a Bayesian perspective, there's nothing wrong with using a sequential analysis. Since Bayesian decision theory generally does not worry about type I errors, there's nothing wrong with multiple peeks. However, in practice, it's a lot more of a gray area. Derived prior distributions don't really capture our knowledge before seeing the data, but we can hand wave this issue away by saying that the likelihood will typically dominate the prior, so this isn't an issue. But if we do a sequential analysis, we may be analyzing the data when we have very little data. Suddenly, miss-specification of the prior becomes a really big issue!

To be clear, I think sequential analyses are a very good idea. But there are downsides.

  • 1
    $\begingroup$ Very well explained. $\endgroup$ – rnso May 14 '19 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.