In my line of research we often calculate prospective power of a proposed study as a way to determine what sample size we should aim for. However, I'm often studying phenomena, or interactions between various phenomena, that have rarely been studied, so it is difficult to get an accurate estimate of what the standardized mean difference should be in the proposed study.
This of course leads to a situation where you're guessing at what your expected difference is, which then leads to inaccurate estimates of desired sample size.
One thing I don't really understand, however, is why it is not statistically sound to do the following:
- Begin running the experiment without a power calculation/estimation of intended sample size
- After you have collected x amount of data, use the observed mean difference in your sample to perform a power calculation, to determine when you should stop running participants
Is this wrong simply because the sample you run may not be representative of the true state of affairs in the population, therefore your observed difference is inaccurate?
And the difference between this approach and using the literature as a guide to determine the effect size is because the literature has already deemed that effect size to be statistically significant, and therefore likely to be representative of the population parameter?
Or is there a deeper statistic understanding that I'm missing here?
Furthermore, what is the typical solution to estimating a desired sample size when you don't have much literature to look at for effect sizes?