# What is the substantive meaning of one statistical test is more powerful than another?

There are some research claims that one statistical test is more powerful than another. For example, a highly cited study states:

Results show that the Shapiro-Wilk test is the most powerful normality test, followed by the Anderson-Darling test, Lilliefors test, and Kolmogorov-Smirnov test.

The general idea of the argument is as follows (please correct me if I'm mistaken or misunderstanding something):

• given a nominal alpha level and alternative hypotheses ($$H_1$$).
• based on alternative hypotheses ($$H_1$$), simulation data is repeatedly generated, and the acceptance rate of $$H_1$$ for different statistical tests is calculated.
• the statistical tests of highest accept rate under alternative hypotheses ($$H_1$$) is most powerful

My confusion stems from this point: they only fix the nominal alpha level, not the actual alpha level (it seems logically impossible to fix actual alpha level under $$H_1$$). For instance, if a statistical test consistently returns a p-value of 0.0(so actual alpha level === 1.0 and power === 1.0), is it always considered the most powerful regardless of the scenario?"

alternate approach to comparing the power of statistical tests, as suggested by @Frank Harrell @:

• verify if the actual and nominal $$\alpha$$ coincide under $$H_0$$.
• compare the power of the statistical tests under $$H_1$$.

but if given the same data-generating process, nominal alpha level, and the nominal alpha equal to actual alpha under $$H_0$$, is it possible for two statistical tests to exhibit different power under $$H_1$$?

in many cases, the distribution under $$H_1$$ is essentially a left or right shift from the distribution under $$H_0$$. If two statistical tests exhibit the same actual alpha for all given nominal alpha levels under $$H_0$$, does this imply that $$H_0$$ distribution of two statistical tests is the same? Furthermore, does this imply that they should consistently demonstrate the same power under $$H_1$$?

reference: Razali, Nornadiah; Wah, Yap Bee (2011). Power comparisons of Shapiro–Wilk, Kolmogorov–Smirnov, Lilliefors and Anderson–Darling tests

• Perhaps I have misunderstood, but "a statistical test consistently returns a p-value of 1.0" suggests that it never rejects the null hypothesis and so has no power Commented Aug 12, 2023 at 22:04
• you are correct, this a typo error, i mean $p = 0.0$, so actual alpha level === 1.0 and power === 1.0
– wei
Commented Aug 13, 2023 at 1:17
• Some things to note: 1. Power for a compound alternative depends on the specific alternative hypothesis being considered (power is not a single value) 2. Power for omnibus goodness of fit tests won't even be a function of some single effect-size parameter. 3. There is no test against such general alternatives that's most powerful for each alternative. 4. Consequently, unless some disclaimers/qualifying statements are added the quoted general claim is simply false. Presumably (/hopefully) the necessary disclaimers/context immediately precede or follow the quote. Commented Aug 13, 2023 at 18:14

If a test has any power and always returns $$p=1.0$$ that implies that the sample size is too small to be doing statistical testing. Power of competing tests is compared at
• the null, to show that the power is $$\alpha$$ so that actual and nominal $$\alpha$$ coincide
Note that test for normality is not best statistical practice, as you are implying a binary decision (e.g., parametric vs nonparametric methods) and are implicitly assuming that the normality test has high power. You are also implying that nonparametric methods don't work well under the Gaussian distribution, which is not the case. Use robust methods with fewer assumptions instead of relying on the data to tell you which model to use. Except with very large $$N$$ the data are often unable to tell you what you need to make the "right" decision.
• Thank you for your response. I still have some confusion. If we consider the same data-generating process, nominal alpha level, and the nominal alpha being equal to the actual alpha under $H_0$, is it possible for two statistical tests to exhibit different power under $H_1$? In many cases, the distribution under $H_1$ is essentially a left or right shift from the distribution under $H_0$. If two statistical tests exhibit the same actual alpha for all given actual alpha levels under $H_0$, does this imply that they should always demonstrate the same power under $H_1$?
• Oh yes it's very possible. $H_1$ can be far from a simple shift. Example: the power will differ much if you compare two samples from Gaussian distributions with different $\sigma$ and the sample sizes differ, if you compare an ordinary $t$-test with a Welch $t$-test. Another example: when comparing two Gaussian samples with the same $\sigma$ using a $t$-test and a Wilcoxon test the power of the Wilcoxon test will be very slightly lower than the power of the $t$-test. Commented Aug 12, 2023 at 15:14