6
$\begingroup$

I'm looking for an analysis (I assume a book or website, but feel free to put it all in comments here) that provides an in-depth discussion of the power/accuracy of normality assessments such as Shapiro-Wilk. In short, I understand that Shapiro-Wilk "is good for small data sets" and with large data sets it is overly sensitive to small deviations from normality - but what does this mean quantitatively?

Overall I'd like to get a feel for something like this: If I have n data points, I'm mm% confident that the Shapiro-Wilk (and/or other tests) test is providing me a reasonable conclusion. How does mm% change when n is 15 ... 10 ... 7? How about as it goes from 100 to 500 ... 1000? Please note that I do believe in the power of graphical assessments as well, but I'm keeping those out of scope for this question and I'm looking for numeric, quantitative assessments in this case.

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Frame challenge: what I would like to see (maybe I will post this as a separate question) is *how well the Shapiro-Wilk statistic {or other tests/statistics} reflects the magnitude of non-robustness to non-Normality. Obviously one would have to specify (1) type of deviation, (2) analysis type (e.g. ANOVA, t-test, regression), (3) sample size/experimental design, and (4) problem of concern (bias, type 1 error/poor coverage, etc.) $\endgroup$
    – Ben Bolker
    Feb 14, 2021 at 21:53
  • 1
    $\begingroup$ @Ben Bolker Sometimes, in a slightly perverse spirit, I show people the effects of running Shapiro-Wilk on all variables in a dataset. Often the strongest rejections are for indicator variables with values 0 and 1. Naturally, this is not wrong: Shapiro-Wilk is doing what it is designed to do and discerning that such distributions are not normal. Equally naturally, it is irrelevant to good practice, as it is impossible for an indicator variable to be normally distributed and that shouldn't matter any way. In practice people care more about outliers, long tails and/or skewness. $\endgroup$
    – Nick Cox
    Feb 15, 2021 at 12:35

3 Answers 3

Reset to default
6
$\begingroup$

What can be simulated is the power of the Shapiro-Wilk test for detecting specific non-normal distributions. This depends strongly on the exact distribution you want to detect. For this reason a plethora of results can be considered, and you can find examples for the test working well and not so well. (This means that data from a certain non-normal distribution with a certain sample size is simulated, and by this you can determine, at least approximately, the probability for the S-W test to reject normality; the higher the better.)

Here is some work that simulates the power of the Shapiro-Wilk test:

Hadi Alizadeh, N. R. Arghami (2011) Monte Carlo comparison of seven normality tests. Journal of Statistical Computation and Simulation 81(8):965-972 https://www.researchgate.net/publication/232942214_Monte_Carlo_comparison_of_seven_normality_tests

Hadi Alizadeh Noughabi (2018) A Comprehensive Study on Power of Tests for Normality. Journal of Statistical Theory and Applications 17. 647 - 660 https://www.atlantis-press.com/journals/jsta/125905578/view

Razali, N. M. & Wah, Y. B. (2011), ‘Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests’, Journal of Statistical Modeling and Analytics 2, 21–33. https://www.nrc.gov/docs/ML1714/ML17143A100.pdf

Danush K. Wijekularathna, Ananda B. W. Manage & Stephen M. Scariano (2019) Power analysis of several normality tests: A Monte Carlo simulation study, Communications in Statistics - Simulation and Computation, DOI: 10.1080/03610918.2019.1658780

Some older work is cited in Thode, H. C. (2002) Testing for Normality. Marcel Dekker Inc.

Improve this answer
$\endgroup$
3
$\begingroup$

You are asking for the probability of a "reasonable conclusion". You can get this if and only if you give a precise enough definition of a reasonable conclusion. I too would love a procedure that reliably tells me when deviation from normality is important enough to matter, but there is a gradation from sample sizes of 1 and 2 -- for which any sample whatsoever is totally consistent with a normal distribution -- to arbitrarily large sample sizes -- for which trivial deviations will return significance levels below conventional thresholds.

What is important enough to matter depends on your purposes and on your view of the data, as statistics is a craft drawing on personal expertise and experience just as much as it is codified technique. For example:

  1. Marginal normal distributions are only rarely required or even ideal.

  2. In practice if I found very close approximations to conditional normal distributions -- e.g. in residuals from models known to be sound scientifically for high quality data -- I would suspect fraud more frequently than I would believe the result implicitly.

It's quite hard to teach beginners how to sit loose to significance testing, not least in disciplines that persist in over-rating it, and because you have to look at many datasets before you start getting your own independent sense of what helps. But the existence proof that beginners can grow into experts is the fact that there are experts.

EDIT A general purpose "test" is to post a normal quantile plot of the data together with several normal quantile plots for random samples of the same size from a normal. Convenient numbers of simulated samples might be 24, 35, 48, ... permitting, when combined with the original, a 5 x 5, 6 x 6, 7 x 7, ... display. This is the line-up test, similar to the idea that a suspect should be shown to witnesses together with various arbitrary people. If the suspect isn't identifiable from the others, the case for being different isn't supported. Although quite often re-invented fairly recently, the idea is in Walter Shewhart's main book (and may yet be older). One attraction of the test is the Anna Karenina principle that each non-normal distribution may be non-normal in its own different way.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I completely agree, I'm looking at this from the standpoint of providing guidance to non-statisticians, but technical people. I would love to provide a large training course, but in this case the guidance must be brief, with an attempt at a standardized approach (I know, I know...). In such a case, thresholds must be determined, or at least discussed from the standpoint of ideal conditions with brief discussion of repercussion of smaller sample sizes. I'd love to say something like "Sample sizes should be 15 units or greater, but groups of 10 may be acceptable with additional scrutiny..." $\endgroup$
    – Porter
    Feb 14, 2021 at 20:20
  • $\begingroup$ I suppose part of your advice could be that people need more data. However, in my experience the usual deal is to make sense of the data that are in hand. (I am a non-statistician, a geographer by training.) $\endgroup$
    – Nick Cox
    Feb 14, 2021 at 21:26
0
$\begingroup$

As you allude to when you bring up the use of graphical examination for this task of assessing normality, the answer is $0\%$ $\forall$ $n$.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I'm afraid you're simply wrong on this one. You're suggesting that Shapiro-Wilk is a random number generator, which is absolutely incorrect. Graphical analysis is important, but numerical analysis is always preferable. $\endgroup$
    – Porter
    Feb 14, 2021 at 14:50
  • 2
    $\begingroup$ @Porter It is not a random number generator. For given input data, it will generate the same p-value every time. $\endgroup$
    – Dave
    Feb 14, 2021 at 15:37
  • 1
    $\begingroup$ A special case of that comment is to say that for given seed, a pseudo random number generator will generate the same value every time. @Porter's is not refuted, is it? $\endgroup$
    – Rui Barradas
    Feb 15, 2021 at 8:39
  • $\begingroup$ I don't think Dave is claiming that Shapiro-Wilk is meaningless, which the "random number" comments imply. He is just saying that as is true with any such test there is no certainty that such a test points out in the direction you prefer: there is always scope for it to fail to detect an "important" departure or for it to indicate a "trivial" one. The answer of @Lewian seems closest to understanding what the OP really wants, formal investigations of power; this answer and mine reacted to the broad wording of the question. $\endgroup$
    – Nick Cox
    Feb 15, 2021 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.