Timeline for Is normality testing 'essentially useless'?
Current License: CC BY-SA 4.0
40 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 6 at 15:26 | history | edited | Nick Cox | CC BY-SA 4.0 |
deleted 1 character in body
|
| Oct 19, 2022 at 18:05 | history | wiki removed | kjetil b halvorsen♦ | ||
| Oct 4, 2022 at 23:48 | comment | added | user67724 | @Joris, I could argue that the QQ plot you showed belongs to one of the 13% of cases (1-0.87) where normality is not rejected by SW test. Hence, it would be helpful to show the QQ plot for two instances, one where the SW test rejects normality and one where it does not. | |
| Jun 26, 2018 at 12:53 | comment | added | Glen_b | The central limit theorem may sometimes be useful when looking at the level of a test but it doesn't help with the power; generally the relative efficiency (compared to say the most powerful test available) doesn't tend to increase with sample size. | |
| S Mar 5, 2018 at 7:03 | history | suggested | DBinJP | CC BY-SA 3.0 |
added introduction to test
|
| Mar 5, 2018 at 3:07 | review | Suggested edits | |||
| S Mar 5, 2018 at 7:03 | |||||
| Dec 14, 2017 at 10:22 | comment | added | Joris Meys | @thc Just use the QQ plot to justify it. And if the sample size is large enough, the central limit theorem provides you with the normality assumption already in many cases. | |
| Dec 13, 2017 at 20:02 | comment | added | thc | @JorisMeys Thanks for your illustrative answer. Your post clearly illustrates the problem, but what is the solution? Is there an "almost normal" test? Something conceptually like a TOST equivalence test? I am facing this exact issue where a reviewer that is asking for justification of normality assumption - the QQ plots look good, but the test is significant due to large sample size. | |
| Jun 16, 2017 at 19:06 | history | edited | Joris Meys | CC BY-SA 3.0 |
as per request in the comments a few years ago, clarified a few things
|
| Jun 16, 2017 at 18:08 | comment | added | DWin | This example could be used as an argument that failing such a "normality test" should be an argument for applying regression or other classification methods (rather than immediately applying a transformation). | |
| Sep 13, 2016 at 21:17 | history | edited | gung - Reinstate Monica | CC BY-SA 3.0 |
light formatting for readability
|
| Sep 4, 2016 at 16:47 | comment | added | Joris Meys | @Milos Even in the original paper the author refered already to the statistic as sensitive, even with small samples (n < 20). It is also sensitive to outliers, according to the same 1965 paper. Also remember that the W statistic has a maximum of 1 (indicating perfect normality) and look at the critical values of W for rejecting the null. At n=10, this is 0.84. At n=50, this is 0.947. So at n=50, a far smaller deviation will be significant. At n=5000, even a W value of 0.999 is highly significant. That's basic statistics. | |
| Sep 3, 2016 at 14:06 | comment | added | Milos | @JorisMeys Could You point me to a paper or a proof that "when n gets large, even the smallest deviation from perfect normality will lead to a significant result"? :) | |
| Oct 24, 2014 at 17:09 | history | edited | Nick Cox | CC BY-SA 3.0 |
Wilk (M.B.), not Wilks (S.S.)
|
| Aug 20, 2014 at 19:35 | history | edited | Tal Galili | CC BY-SA 3.0 |
edited body
|
| Nov 6, 2013 at 14:36 | comment | added | Joris Meys | @whuber You're free to update :) otherwise I'll update it when I find a bit more time. Cheers. | |
| Oct 29, 2013 at 18:22 | comment | added | whuber♦ | I don't disagree with you; I am only (mildly) objecting that the important points you have recently made in these comments did not appear in your answer. | |
| Oct 25, 2013 at 16:03 | comment | added | Joris Meys | Btw, QQ plots are not meant to detect such mixtures. They're graphical tools that give you a fair idea about whether or not you'll lose power an even get biased estimates when using specific tests. That's all there is to them. For 99% of the statistical questions in practical science, that's more than enough. | |
| Oct 25, 2013 at 16:03 | comment | added | Joris Meys | Not one real life distribution is perfectly normal. So with large enough samples, all normality test should reject the null. So yes, SW does what it needs to do. But it is worthless for applied statistics. There's no point in going to eg a Wilcoxon when having a sample size of 5000 and an almost normal distribution. And that's what OP's remark was all about: does it make sense to test for normality when having large sample sizes? Answer: no. Why? because you detect (correctly) a deviation that doesn't matter for your analysis. As pointed out by the QQ plots | |
| Oct 25, 2013 at 14:17 | comment | added | whuber♦ | I had relied on what you wrote and misunderstood what you meant by an "almost-Normal" distribution. I now see--but only by reading the code and carefully testing it--that you are simulating from three standard Normal distributions with means at $0,$ $1,$ and $2$ and combining the results in a $2:2:1$ ratio. Wouldn't you hope that a good test of Normality would reject the null in this case? What you have effectively demonstrated is that QQ plots are not very good at detecting such mixtures, that's all! | |
| Oct 25, 2013 at 12:32 | comment | added | Joris Meys | That fact is true, but has no bearance with the CLT. The CLT is pretty specific about under what conditions the approximation holds. You throw different things on the same heap. Yes, Wilcox gives those examples. No, he isn't talking about large sample sizes or dismissing the CLT, far from even. He rightfully points out people forget about the conditions under which the CLT holds. I agree with you that testing differences with a sample size of 5000 doesn't make sense without stating what the minimal relevant difference is. But that's a whole other issue. | |
| Oct 25, 2013 at 12:05 | comment | added | Frank Harrell | The fact that often we can't tell from a sample whether that sample can adequately be analyzed by a normality-assuming method is enough for me. And Wilcox gives examples where the non-normality (contamination of a normal distribution with another normal distribution with higher variance) is so imperceptible that you cannot see it in the density function, yet the tiny bit of non-normality causes a major distortion in tests' operating characteristics. Another issue that most statisticians have not really addressed is that the standard deviation may not be meaningful with asymmetry. | |
| Oct 25, 2013 at 9:45 | comment | added | Joris Meys | @FrankHarrell I fail to see your point. Rand Wilcox was talking about sample sizes of 30 and more. The question is about very large samples. 30 isn't even large. 5000, that's large (and not that large actually). Doing the math Rand Wilcox did, the variance of the mean follows the chi-squared distribution pretty well for a sample of 5000, even when originating from a pretty skewed distribution. | |
| Oct 25, 2013 at 9:36 | comment | added | Joris Meys | @whuber This answer addresses the question. The whole point of the question is the "near" in "near-normality". S-W tests what is the chance that the sample is drawn from a normal distribution. As the distributions I constructed are deliberately not normal, you'd expect the S-W test to do what it promises: reject the null. The whole point is that this rejection is meaningless in large samples, as the deviation from normality does not result in a loss of power there. So the test is correct, but meaningless, as shown by the QQplots | |
| Oct 24, 2013 at 21:16 | comment | added | whuber♦ |
This answer appears not to address the question: it merely demonstrates that the S-W test does not achieve its nominal confidence level, and so it identifies a flaw in that test (or at least in the R implementation of it). But that's all--it has no bearing on the scope of usefulness of normality testing in general. The initial assertion that normality tests always reject on large sample sizes is simply incorrect.
|
|
| Aug 1, 2013 at 11:42 | comment | added | Frank Harrell | @joris-meys the central limit theorem does not help unless the population standard deviation is known. Very tiny disturbances in the random variable can distort the sample variance and make the distribution of a test statistic very far from the $t$ distribution, as shown by Rand Wilcox. | |
| Mar 19, 2013 at 17:08 | comment | added | Joris Meys |
@maximus with the function qqnormin R
|
|
| Mar 17, 2013 at 10:03 | comment | added | Le Max | Wow thx for your answer! How did you draw the qqplots? | |
| Feb 4, 2012 at 14:24 | comment | added | probabilityislogic | This is another example of why p-values need to move down as the sample size goes up. 0.05 is not stringent enough in big data world. Just my curiosity - what happens if you set the pvalue to depend on sample size? | |
| Feb 11, 2011 at 8:46 | comment | added | posdef | @joris: I see, thanks for straightening it out for me :) | |
| Feb 10, 2011 at 15:02 | comment | added | Joris Meys | Indeed. 0.87 is the proportion of datasets that give a deviation from normality, meaning that in 87% of the datasets from an almost normal distribution, Shapiro-Wilks will have a p-value smaller than 0.05. The second part is just an example of some datasets that illustrate this. | |
| Feb 10, 2011 at 14:58 | comment | added | posdef | @joris: I think there might have been a misunderstanding; Shapiro-Wilks give p_{n5000} = 0.87 while the second calculation yields p_{n5000} = 0.007. Or have I misunderstood something? | |
| Feb 10, 2011 at 13:31 | comment | added | Joris Meys | @posdef : those are just the p-values of the shapiro-wilks test, to indicate that they contradict the qq-plots. | |
| Feb 10, 2011 at 13:04 | comment | added | posdef | +1: great answer, very intuitive. Perhaps a bit off-topic but how would one go about implement the second method without qq-plots (due to lack of visualization)? What logical steps are taken here to get the p-values? | |
| Sep 10, 2010 at 17:21 | vote | accept | shabbychef | ||
| Sep 9, 2010 at 9:37 | comment | added | Dikran Marsupial | yes, the real question is not whether the data are actually distributed normally but are they sufficiently normal for the underlying assumption of normality to be reasonable for the practical purpose of the analysis, and I would have thought the CLT based argument is normally [sic] sufficient for that. | |
| Sep 8, 2010 at 23:19 | comment | added | Joris Meys | On a side note, the central limit theorem makes the formal normality check unnecessary in many cases when n is large. | |
| Sep 8, 2010 at 22:35 | comment | added | shabbychef | this is great! I'm slapping myself for not doing the experiments myself... | |
| Sep 8, 2010 at 22:29 | history | edited | Joris Meys | CC BY-SA 2.5 |
added 153 characters in body
|
| Sep 8, 2010 at 22:23 | history | answered | Joris Meys | CC BY-SA 2.5 |