There are some research claims that one statistical test is more powerful than another. For example, a highly cited study states:
Results show that the Shapiro-Wilk test is the most powerful normality test, followed by the Anderson-Darling test, Lilliefors test, and Kolmogorov-Smirnov test.
The general idea of the argument is as follows (please correct me if I'm mistaken or misunderstanding something):
- given a nominal alpha level and alternative hypotheses ($H_1$).
- based on alternative hypotheses ($H_1$), simulation data is repeatedly generated, and the acceptance rate of $H_1$ for different statistical tests is calculated.
- the statistical tests of highest accept rate under alternative hypotheses ($H_1$) is most powerful
My confusion stems from this point: they only fix the nominal alpha level, not the actual alpha level (it seems logically impossible to fix actual alpha level under $H_1$). For instance, if a statistical test consistently returns a p-value of 0.0(so actual alpha level === 1.0 and power === 1.0), is it always considered the most powerful regardless of the scenario?"
alternate approach to comparing the power of statistical tests, as suggested by @Frank Harrell @:
- verify if the actual and nominal $\alpha$ coincide under $H_0$.
- compare the power of the statistical tests under $H_1$.
but if given the same data-generating process, nominal alpha level, and the nominal alpha equal to actual alpha under $H_0$, is it possible for two statistical tests to exhibit different power under $H_1$?
in many cases, the distribution under $H_1$ is essentially a left or right shift from the distribution under $H_0$. If two statistical tests exhibit the same actual alpha for all given nominal alpha levels under $H_0$, does this imply that $H_0$ distribution of two statistical tests is the same? Furthermore, does this imply that they should consistently demonstrate the same power under $H_1$?
reference: Razali, Nornadiah; Wah, Yap Bee (2011). Power comparisons of Shapiro–Wilk, Kolmogorov–Smirnov, Lilliefors and Anderson–Darling tests