5
$\begingroup$

So, I'm actually a biologist trying to wrap my head around the idea of power of analysis to help design an experiment with the proper sample size. I understand that power of analysis is used to help avoid type II errors, but I came across this paper:

http://www.benthamscience.com/open/toepij/articles/V003/16TOEPIJ.pdf

which seems to say (if I'm not mistaken) that in underpowered studies, you also increase your risk for false positives. The other thing that confuses me, is that if you keep adding more samples to increase your power, of course you make any finding statistically significant, even if your effect is small. Is there some sort of balance between power and number of samples so that one avoids the pitfalls of an underpowered experiment, and also an overpowered experiment where the results will produce a statically significant result that might not be interesting because the effect size is so small?

Improve this question
$\endgroup$
2
  • $\begingroup$ If you present standardized effect sizes, the last issue goes away - the reader can see that it was significant, but also that the effect was small, and interpret the findings accordingly. $\endgroup$
    – atrichornis
    Commented Jul 26, 2013 at 10:06
  • 1
    $\begingroup$ No, don't use standardized effect sizes for that purpose. You need to state effects in biologically relevant units. At the heart of the "large sample sizes will detect irrelevant differences" problem, along with multiplicity issues, is the use of null values for evidence assessment. The likelihood and Bayesian paradigms have solutions for this problem. $\endgroup$
    – Frank Harrell
    Commented Jul 26, 2013 at 12:09

1 Answer 1

Reset to default
1
$\begingroup$

From a quick skim it seems like they're basically taking a Bayesian viewpoint and computing a particular probability (H0 true|reject if I understood what they were getting at) that they argue must go up as the sample size goes down; if that's what the claim was, then as far as that goes, it's valid, because the denominator in Bayes rule must decrease as the sample size goes down while the significance level and P(H0 true) are presumably fixed.

A frequentist would argue that their rate of reject|H0 true is fixed, and they'd likely say that's what they care about.

On the gripping hand, in the overwhelming majority of studies, the true overall rate of false&positives must be effectively zero at every sample size, since in most circumstances nulls are simply not exactly true.

I guess it comes down to what probability you want.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.