5
$\begingroup$

There are some research claims that one statistical test is more powerful than another. For example, a highly cited study states:

Results show that the Shapiro-Wilk test is the most powerful normality test, followed by the Anderson-Darling test, Lilliefors test, and Kolmogorov-Smirnov test.

The general idea of the argument is as follows (please correct me if I'm mistaken or misunderstanding something):

  • given a nominal alpha level and alternative hypotheses ($H_1$).
  • based on alternative hypotheses ($H_1$), simulation data is repeatedly generated, and the acceptance rate of $H_1$ for different statistical tests is calculated.
  • the statistical tests of highest accept rate under alternative hypotheses ($H_1$) is most powerful

My confusion stems from this point: they only fix the nominal alpha level, not the actual alpha level (it seems logically impossible to fix actual alpha level under $H_1$). For instance, if a statistical test consistently returns a p-value of 0.0(so actual alpha level === 1.0 and power === 1.0), is it always considered the most powerful regardless of the scenario?"

alternate approach to comparing the power of statistical tests, as suggested by @Frank Harrell @:

  • verify if the actual and nominal $\alpha$ coincide under $H_0$.
  • compare the power of the statistical tests under $H_1$.

but if given the same data-generating process, nominal alpha level, and the nominal alpha equal to actual alpha under $H_0$, is it possible for two statistical tests to exhibit different power under $H_1$?

in many cases, the distribution under $H_1$ is essentially a left or right shift from the distribution under $H_0$. If two statistical tests exhibit the same actual alpha for all given nominal alpha levels under $H_0$, does this imply that $H_0$ distribution of two statistical tests is the same? Furthermore, does this imply that they should consistently demonstrate the same power under $H_1$?

reference: Razali, Nornadiah; Wah, Yap Bee (2011). Power comparisons of Shapiro–Wilk, Kolmogorov–Smirnov, Lilliefors and Anderson–Darling tests

Improve this question
$\endgroup$
3
  • $\begingroup$ Perhaps I have misunderstood, but "a statistical test consistently returns a p-value of 1.0" suggests that it never rejects the null hypothesis and so has no power $\endgroup$
    – Henry
    Aug 12 at 22:04
  • $\begingroup$ you are correct, this a typo error, i mean $p = 0.0$, so actual alpha level === 1.0 and power === 1.0 $\endgroup$
    – wei
    Aug 13 at 1:17
  • $\begingroup$ Some things to note: 1. Power for a compound alternative depends on the specific alternative hypothesis being considered (power is not a single value) 2. Power for omnibus goodness of fit tests won't even be a function of some single effect-size parameter. 3. There is no test against such general alternatives that's most powerful for each alternative. 4. Consequently, unless some disclaimers/qualifying statements are added the quoted general claim is simply false. Presumably (/hopefully) the necessary disclaimers/context immediately precede or follow the quote. $\endgroup$
    – Glen_b
    Aug 13 at 18:14

1 Answer 1

Reset to default
7
$\begingroup$

If a test has any power and always returns $p=1.0$ that implies that the sample size is too small to be doing statistical testing. Power of competing tests is compared at

  • the null, to show that the power is $\alpha$ so that actual and nominal $\alpha$ coincide
  • non-null values that are not trivial, e.g., estimate the power of method 2 when the power of method 1 is 0.5, 0.6, 0.7, 0.8, 0.9

Note that test for normality is not best statistical practice, as you are implying a binary decision (e.g., parametric vs nonparametric methods) and are implicitly assuming that the normality test has high power. You are also implying that nonparametric methods don't work well under the Gaussian distribution, which is not the case. Use robust methods with fewer assumptions instead of relying on the data to tell you which model to use. Except with very large $N$ the data are often unable to tell you what you need to make the "right" decision.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your response. I still have some confusion. If we consider the same data-generating process, nominal alpha level, and the nominal alpha being equal to the actual alpha under $H_0$, is it possible for two statistical tests to exhibit different power under $H_1$? In many cases, the distribution under $H_1$ is essentially a left or right shift from the distribution under $H_0$. If two statistical tests exhibit the same actual alpha for all given actual alpha levels under $H_0$, does this imply that they should always demonstrate the same power under $H_1$? $\endgroup$
    – wei
    Aug 12 at 14:03
  • $\begingroup$ Oh yes it's very possible. $H_1$ can be far from a simple shift. Example: the power will differ much if you compare two samples from Gaussian distributions with different $\sigma$ and the sample sizes differ, if you compare an ordinary $t$-test with a Welch $t$-test. Another example: when comparing two Gaussian samples with the same $\sigma$ using a $t$-test and a Wilcoxon test the power of the Wilcoxon test will be very slightly lower than the power of the $t$-test. $\endgroup$
    – Frank Harrell
    Aug 12 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.