# What does statistical power mean when we are interested in the probability of correctly not rejecting the null hypothesis?

It seems that statisticians are usually only interested in statistical power. In other words, they are interested in the probability of correctly rejecting the null hypothesis.

• What if we are interested in the probability of correctly not rejecting the null hypothesis?
• Would we change the hypothesis test so that we are computing a power?

The probability of "correctly not rejecting the null hypothesis"--i.e., if the null hypothesis is true, we do not reject it--is controlled by the significance level at which we are doing the test. If I choose a significance level of $\alpha = .05$, so that I reject if my $p$-value is less than .05, then my probability of correctly not rejecting the null hypothesis is $1-.05 = .95$. If rejecting the null hypothesis when it is in fact true would have very bad consequences then we might use a smaller $\alpha$, say .01 or even .001, which gives us a higher probability of correctly not rejecting the null hypothesis.

So, in fact, we already control this probability--in fact, this is much easier to control than the power. Because it's much easier there's much less discussion about it, which is probably why you concluded that statisticians aren't interested in it.

• So a 95% confidence interval indicates that the probability of correctly not rejecting the null hypothesis is 95%? – proton Jan 22 '13 at 15:00
• @proton 95% confidence intervals aren't a hypothesis test... – Jonathan Christensen Jan 22 '13 at 16:18
• I thought confidence intervals and hypothesis tests were equivalent? – proton Jan 22 '13 at 16:26
• There is often (but not always) a sort of duality between confidence intervals and hypothesis tests, but they aren't equivalent. There are settings--such as F tests in multiple regression--where there's a well-defined hypothesis test but not a meaningful confidence interval related to it. When the relationship exists (such as simple t-tests), your statement is correct: we reject the null hypothesis at $\alpha=.05$ when it lies outside the corresponding 95% confidence interval, and there is a 95% probability of correctly not rejecting the null hypothesis. – Jonathan Christensen Jan 22 '13 at 16:30
• +1 for a good answer. Regarding your comment above, note that it is possible to compute a non-central F distribution, & from that, calculate a 95% CI (which would be a 1-sided CI in this case). It would be complicated & unintuitive, & I don't know if there's software to do it for you (there may be some obscure R package), so I doubt anyone ever does it, but it is possible. – gung - Reinstate Monica Mar 11 '13 at 16:58

I think you are interested in equivalence testing. See this other question on testing a hypothesis of no group differences.

There are various approaches that can be adopted to assess whether the null hypothesis is true. In general, the absence of statistically significant effect is very week evidence for the truth of the null hypothesis.

Three common approaches include (a) looking at confidence intervals; (b) looking at bayesian posterior densities on the parameter of interest; or (c) setting up two one-sided significance tests.

The confidence interval and Bayesian posterior density approach are often used to quantify uncertainty of a parameter of interest. The Bayesian approach is arguably more aligned with the question of interest where the parameter is seen as unknown. Looking at such intervals you could judge that if the interval includes the null hypothesis and other plausible values are sufficiently close to zero, then this means that the null hypothesis or something sufficiently similar is the most likely the truth.

A similar approach is to set up two one-sided significance tests. E.g., when testing whether the means are the same for two groups ($\delta = \frac{\mu_1 - \mu_2}{\sigma}$) you could test whether $\delta$ is significantly less than .1 and significantly more than -.1. In this cases you could calculate the statistical power of such tests assuming:

• the null hypothesis is true
• alpha
• sample size

Or if you wanted to hold power constant, then you could assess what sample size would be required. You could also vary the threshold for equivalence and see how as you expand the width of the equivalence threshold your power increases.

This is a common applied problem in the context of equivalence and non-inferiority testing for drugs (e.g., Walker and Nowacki, 2011).