3
$\begingroup$

My understanding of statistical power is that it is the likelihood of correctly rejecting the null hypothesis, with low power meaning you are very likely to make a beta error (failing to reject the null when the alternative hypothesis is in fact true). Therefore it seems to me that if you reject the null, you, by definition, had enough statistical power. You might have made an alpha error, and incorrectly rejected the null, but that's only indirectly related to your power.

So, of you reject the null, does it make sense to ask whether you had enough statistical power?

$\endgroup$
1
$\begingroup$

Ideally you ask the question about statistical power beforehand, when you design the experiment. In the same way as you make some decision about a useful boundary for the significance level in the hypothesis test, before you do the test. If you do it afterwards then it easily becomes cherry-picking.

In practice people may sometimes wonder about statistical power only after they designed and performed their experiment and statistical tests. Often this occurs when they are not happy with their non-significant result and they try to find ways out, or they discuss their results with critical colleagues that remark that the not rejected null-hypothesis sounds nice, but is meaningless in relation to power.

Do not perform an experiment/hypothesis test if it does not have high enough power. It will not give you much useful information. You should think about this before you do the test in order to prevent practices like cherry-picking or spending lots of time with vague conclusions with weak data sets.

  • If such a low power test ends up in no rejection of the hypothesis then it is not a strong confirmation of the null hypothesis since there is little power to lead to rejection if there is some effect
  • If such a low power test ends up in rejection of the hypothesis then you are still not very sure whether this was because there is indeed an effect and the null-hypothesis is indeed incorrect or because of the probability of a type I error.

Results from frequentist methods are after all, in the end, sort of interpreted in a Bayasian way. The boundaries for confidence intervals, alpha-levels, required power, are all a bit arbitrary when they become instead of 'indicators of the likelihood of the data given some hypothesis' some 'decision rule'.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.