In my line of research we often calculate prospective power of a proposed study as a way to determine what sample size we should aim for. However, I'm often studying phenomena, or interactions between various phenomena, that have rarely been studied, so it is difficult to get an accurate estimate of what the standardized mean difference should be in the proposed study.

This of course leads to a situation where you're guessing at what your expected difference is, which then leads to inaccurate estimates of desired sample size.

One thing I don't really understand, however, is why it is not statistically sound to do the following:

  1. Begin running the experiment without a power calculation/estimation of intended sample size
  2. After you have collected x amount of data, use the observed mean difference in your sample to perform a power calculation, to determine when you should stop running participants

Is this wrong simply because the sample you run may not be representative of the true state of affairs in the population, therefore your observed difference is inaccurate?

And the difference between this approach and using the literature as a guide to determine the effect size is because the literature has already deemed that effect size to be statistically significant, and therefore likely to be representative of the population parameter?

Or is there a deeper statistic understanding that I'm missing here?

Furthermore, what is the typical solution to estimating a desired sample size when you don't have much literature to look at for effect sizes?


The problem is that when things look "good" by chance, you collect less data and are less likely to overturn the lucky chance finding, while if things look"bad", you collect a lot more data to overturn that. As a result you get type 1 error inflation, biased estimates etc.

Adaptive designs (doing what you describe, but then modifying the final analysis to account for doing it) and group sequential designs (taking repeated looks at the data to see whether an effect has already been established and further days ate not needed - again this requires adaptation of the analyses) are some of the attempts to a avoid these issues, while achieving what you want to do.

  • $\begingroup$ When you say "look good" do you mean my subjective assessment of the results mean I would not collect much more data and risk losing the good finding? Or do you mean it in a statistical way, where if p < 0.05, then the power calculations estimated sample size will be lower than if things looked bad? The situation I'm describing is that I will run 50 people right off the bat, use the mean difference found and use that to estimate a desired sample. Then run that entire sample before analyzing the data fully $\endgroup$
    – Simon
    Jan 20 '16 at 7:38
  • 1
    $\begingroup$ I meant "good" in the sense of a large effect size resulting in a low additional sample size being calculated as being needed as opposed to "bad" = a small effect size needing a very large sample size. If you run 50 people, use the observed effect size to calculate the additional needed patients and then analyze the data as if you had not done that, then you will have biased final estimates, invalid p-values and confidence intervals that do not have their nominal coverage. $\endgroup$
    – Björn
    Jan 20 '16 at 8:12
  • $\begingroup$ I just realized that the Statistics in Medicine has just published "Twenty-five years of confirmatory adaptive designs: opportunities and pitfalls" (onlinelibrary.wiley.com/doi/10.1002/sim.6472/full), which may provide a nice introduction to adaptive designs. $\endgroup$
    – Björn
    Jan 20 '16 at 8:12

You don't need to be able to estimate the standardized mean difference. What you need to specify is the smallest value of the (standardized) mean difference that you consider to be of practical importance. This then becomes your alternative hypothesis that you enter into sample size determination software to determine the sample size


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.