17
$\begingroup$

What does it mean for a study to be over-powered?

My impression is that it means that your sample sizes are so large that you have the power to detect minuscule effect sizes. These effect sizes are perhaps so small that they are more likely to result from slight biases in the sampling process than a (not necessarily direct) causal connection between the variables.

Is this the correct intuition? If so, I don't see what the big deal is, as long as the results are interpreted in that light and you manually check and see whether the estimated effect size is large enough to be "meaningful" or not.

Am I missing something? Is there a better recommendation as to what to do in this scenario?

$\endgroup$
2
  • $\begingroup$ Sounds exactly like my intuitive understanding of this term. $\endgroup$
    – Henrik
    Apr 5, 2011 at 19:59
  • $\begingroup$ It is not, because slight biases in the sampling process get reduced with larger sample sizes (law of large numbers). In terms of confidence intervals, the effect size gets closer to 0 in proportion to the shortening of the tails of the confidence intervals. The example above holds for a trial where there is truly no difference in the population $\endgroup$
    – astralwolf
    Mar 6, 2023 at 1:06

4 Answers 4

17
$\begingroup$

I think that your interpretation is incorrect.

You say "These effect sizes are perhaps so small as are more likely result from slight biases in the sampling process than a (not necessarily direct) causal connection between the variables" which seems to imply that the P value in an 'over-powered' study is not the same sort of thing as a P value from a 'properly' powered study. That is wrong. In both cases the P value is the probability of obtaining data as extreme as those observed, or more extreme, if the null hypothesis is true.

If you prefer the Neyman-Pearson approach, the rate of false positive errors obtained from the 'over-powered' study is the same as that of a 'properly' powered study if the same alpha value is used for both.

The difference in interpretation that is needed is that there is a different relationship between statistical significance and scientific significance for over-powered studies. In effect, the over-powered study will give a large probability of obtaining significance even though the effect is, as you say, miniscule, and therefore of questionable importance.

As long as results from an 'over-powered' study are appropriately interpreted (and confidence intervals for the effect size help such an interpretation) there is no statistical problem with an 'over-powered' study. In that light, the only criteria by which a study can actually be over-powered are the ethical and resource allocation issues raised in other answers.

$\endgroup$
3
  • 1
    $\begingroup$ Thanks, this is very informative. I understand that the p-value definition doesn't change. Certainly from a statistical standpoint, the rate of type I errors does not increase. $\endgroup$ Apr 6, 2011 at 16:15
  • 1
    $\begingroup$ By definition, we are fixing the type I error rate in setting the p-value threshold. However, it seems like the difference between "statistical" and "practical" significance is the issue here. When the sample size is able to detect differences much finer than the expected effect size, a difference that is correctly statistically distinct is not practically meaningful (and from the perspective of the "end-user" this is effectively a "false positive" even if it's not a statistical one). However, as you say this starts to get outside the realm of statistics. $\endgroup$ Apr 6, 2011 at 16:33
  • 1
    $\begingroup$ i.e. i think i agree - "the difference in interpretation that is needed is that there is a different relationship between statistical significance and scientific significance" $\endgroup$ Apr 7, 2011 at 2:04
6
$\begingroup$

In medical research trials may be unethical if they recruit too many patients. For example if the goal is to decide which treatment is better it's not ethical any more to treat patients with the worse treatment after it was established to be inferior. Increasing the sample size would, of course, give you a more accurate estimate of the effect size, but you may have to stop well before the effects of factors like "slight biases in the sampling process" appear.

It may also be unethical to spend public money of sufficiently confirmed research.

$\endgroup$
2
$\begingroup$

Everything you've said makes sense (although I don't know what "big deal" you're referring to), and I esp. like your point about effect sizes as opposed to statistical significance. One other consideration is that some studies require the allocation of scarce resources to obtain the participation of each case, and so one wouldn't want to overdo it.

$\endgroup$
1
  • $\begingroup$ Sorry, "big deal" is too much of an editorial comment. The question of whether it's a "bigger deal" than I'm making it out to be is basically a question of whether there are additional considerations of which I may be ignorant. $\endgroup$ Apr 5, 2011 at 22:21
0
$\begingroup$

My experience comes from A/B experiments online, where the issue is usually underpowered studies or measuring the wrong things. But it seems to me an overpowered study produces narrower confidence intervals than comparable studies, lower p-values, and possibly different variance. I imagine this can make it harder to compare similar studies. For example, if I repeated an overpowered study using proper power, my p-value would be higher even if I exactly replicated the effect. Increased sample size can even out variability or introduce variability if there are outliers which might have a higher chance of showing up in a larger sample.

Also, my simulations show that effects other than the ones you are interested in might become significant with a larger sample. So, while the p-value correctly tells you the probability that your results are real, they could be real for reasons other than what you think e.g., a combination of chance, some transient effect you didn't control for, and perhaps some other smaller effect you introduced without realizing it. If the study is just a bit overpowered, the risk of this is low. The problem is often it's hard to know the adequate power e.g., if the baseline metrics and minimum target effect are guesses or turn out different than expected.

I've also come across an article that argues that too large of a sample can make a goodness-of-fit test too sensitive to inconsequential deviations, leading to potentially counter-intuitive results.

That said, I believe it best to err on the side of high rather than low power.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.