If we have some test, and we decrease its level, would the power be expected to increase? I have given some thought to this question before but haven't been able to convince myself of the correct answer as the calculation of power seems to be multi-step and so I cannot make a direct judgement call as I would if we had something like an inverse function. Any help would be greatly appreciated!
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
3
The likelihood of making a type 1 error v. a type 2 error is inversely proportional. Thus, if you make your rejection of the null less stringent, all else being equal, the power of your test should increase.
From Wikipedia on Statistical Power: “One easy way to increase the power of a test is to carry out a less conservative test by using a larger significance criterion, for example 0.10 instead of 0.05. This increases the chance of rejecting the null hypothesis (i.e. obtaining a statistically significant result) when the null hypothesis is false, that is, reduces the risk of a Type II error (false negative regarding whether an effect exists).”
answered Mar 27, 2014 at 14:47
-
$\begingroup$ so just to make sure, what i wrote was false? that if the level is decreased, the power should decrease as well? $\endgroup$ Commented Mar 27, 2014 at 15:11
-
$\begingroup$ @user123276 your statement is generally true. $\endgroup$– user31668Commented Mar 27, 2014 at 15:13
-
$\begingroup$ but decreasing the level of a test, does that mean making $\alpha$ larger or smaller? thanks $\endgroup$ Commented Mar 27, 2014 at 15:16
$\begingroup$
$\endgroup$
The level is of a test is its significance level, $\alpha$. Decreasing the level means making $\alpha$ smaller.
If you make $\alpha$ smaller, you make $\beta$ larger* ... and power is $1-\beta$.
What you're doing when you lower $\alpha$ is moving the critical value further out into the tail, so you make it harder to reject whether $H_0$ is true or false, since by moving the critical value further into the tail, you have reduced the set of potential samples that are in the rejection region.
It's easy to see this 'lockstep' reduction in probability of rejection under $H_0$ and $H_1$ ($\alpha$ and $1-\beta$ respectively) in the context of a power curve, for example, with a one-sample t-test. If you draw the power against the difference between the true population mean and the hypothesized population mean, $\delta=\mu-\mu_0$, you get a curve that is at $\alpha$ when $\delta=0$ and increases as $\delta$ gets further away from 0. If you reduce $\alpha$ by pushing the critical value further into the tail, you "pull the curve down", because you have eliminated some possible** sample arrangements that would previously have led to rejection:
As we move from blue (10% significance level) to dark red (5%) to green (1%), ceteris paribus, the whole power curve moves down.
* nearly always with typical sorts of tests, but it is possible to construct cases where this doesn't necessarily happen
** whether $|\delta|$ was $0$, or small, or large, those possible values of the statistic between the old critical value and the new no longer count as rejections, so each possible value for $\delta$ has a lower rejection rate.
$\begingroup$
$\endgroup$
1
Ceteris paribus, when you decrease the significance level $\alpha$ in a classical hypothesis test, you are increasing the amount of evidence required to reject the null hypothesis. This means that you are less likely to reject the null hypothesis, which lowers the probability of a Type I error, but also reduces the power of your test.
-
$\begingroup$ Although these statements are usually true, they are not necessary consequences of the theory. It is possible for the power for subsets of the alternative hypothesis to improve as $\alpha$ decreases. $\endgroup$– whuber ♦Commented May 13, 2022 at 19:03