Suppose that we choose the size of the test to be $\alpha = 0.05$, and based on our sample size $n$ and magnitude of the absolute value of the estimate, we determine that the test's power (i.e., 1 - Prob(Type 2 Error)) is $99.9\%$.

If I then fail to reject $H_0=0$ against $H_a \not= 0$, what consequences does this have for our belief that $H_0$ is correct, assuming that we have the correct null hypothesis distribution of the test statistic?

What if the power was $50\%$?

  • 2
    $\begingroup$ Have you looked into Bayesian inference as an alternative approach? It might have a stronger intuitive fit with your approach. $\endgroup$ – Peter Ellis Dec 18 '12 at 4:21
  • $\begingroup$ @PeterEllis Haven't yet, I'll do that! $\endgroup$ – Jase Dec 18 '12 at 4:25

When you calculate power it is for a specific alternative value, so failing to reject the null could mean that the null is true, or it could be that it is false, just not as strong as the alternative value.

What is much more meaningful is to look at the confidence interval to see what the plausible/reasonable values the true parameter could be.

Often it is best to think about not just about what the null value would be but what the region of values would be that are practically equivalent to the null (not equal to the null, but close enough that we would not care) vs. the region of practical importance. Then to see where the confidence interval lies. Even if the interval does not include the null, but is completely in the region of "who cares" then that tells us something. If the interval only includes values of interest then that tells us something else. The big problem comes when the interval contains both values of practical importance and the null value, then our results are indeterminant, the result could be nothing or it could be important. Better than worrying about power for a specific alternative is to design the study so that the confidence interval is too narrow to include both the null value and the smallest important difference (this will result in high power, but is a better way to think of things).

  • $\begingroup$ Thank you. I've made a slight edit which may impact upon the validity of your answer? Not sure. $\endgroup$ – Jase Dec 18 '12 at 4:05
  • 1
    $\begingroup$ It does not really change the answer. First what are you hypothesizing to be equal or not equal to 0? it is not the hypothesis, but rather parameters such as the difference of 2 means or proportions. And still the power is calculated assuming a given value (suspected difference, minimally interesting difference, etc.). If you have high power to reject at a given effect size and don't reject then that is suggestive that the effect size is smaller, but a confidence interval still conveys this better when possible. $\endgroup$ – Greg Snow Dec 18 '12 at 16:19
  • $\begingroup$ @GregSnow What if it is not feasible to increase sample size and narrow the CI (expensive study, etc) - with high power, can we be more confident about 'accepting' the null if we believe, say, the minimally interesting difference parameter used in the power calculation? $\endgroup$ – kirk Aug 8 '13 at 9:47
  • $\begingroup$ @kirk, if by "accepting" the null you mean that you accept that the parameter of interest is the exact value in the null, then No (it is really hard to prove exact equality), if on the other hand you mean that you accept that the parameter is in the range of close enough to the null to not care about it, then maybe/probably, but if a confidence interval is computable, then that will demonstrate the idea much better. $\endgroup$ – Greg Snow Aug 12 '13 at 16:06

I will answer this question from the Bayesian perspective. Consider Bayes' theorem: $$ P(H_0\mid S) = \frac{P(S \mid H_0)}{P(S)}P(H_0) $$ where $S$ is a significant result. We can write an analogous expression for $H_A$. Then, dividing the two expressions, we get $$ \frac{P(H_A\mid S)}{P(H_0\mid S)} = \frac{P(S\mid H_A)}{P(S\mid H_0)}\times\frac{P(H_A)}{P(H_0)} $$ or $$ \mbox{Posterior odds} = \frac{\mbox{Power}}{\mbox{Type I error rate}}\times\mbox{Prior odds}. $$

The term in the middle is called the Bayes factor, and represents the evidence in the data, or the effect on our belief. In particular, the effect of a significant result on our belief (in terms of relative odds of the alternative to the null) is exactly the power over the type I error rate, IF we only know that the result is significant (other facts, for instance the exact $p$ value, might change this assessment). This a significant result with $\alpha=0.05$ and power of 0.8 has an effect of .8/.05=16 on our relative beliefs.

Now for a nonsignificant effect, $N$, $$ \frac{P(H_0\mid N)}{P(H_A\mid N)} = \frac{P(N\mid H_0)}{P(N\mid H_A)}\times\frac{P(H_0)}{P(H_A)} $$ which is $$ \mbox{Posterior odds} = \frac{1-\alpha}{\beta}\times\mbox{Prior odds}. $$ The Bayes factor is thus $(1-\alpha)/\beta$. Again, this is the effect that the nonsignificant result has on our (relative) beliefs, IF all we know is that the result is nonsignificant.

Taking your example numbers, if $\alpha=.05$ and $1-\beta=.999$, then $$ \frac{1-\alpha}{\beta} = \frac{.95}{.001} = 950, $$ that is, the relative evidence in favor of the null hypothesis is enough to shift your odds by a factor of 950. This is extremely strong evidence.

If the power is .5, then $$ \frac{1-\alpha}{\beta} = \frac{.95}{.5} = 1.9, $$ which is extremely weak evidence (keep in mind 1.0 is no evidence either way).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.