# Is normality testing 'essentially useless'?

A former colleague once argued to me as follows:

"we usually apply normality tests to the results of processes that, under the null, generate random variables that are only asymptotically or nearly normal (with the 'asymptotically' part dependent on some quantity which we cannot make large); In the era of cheap memory, big data, and fast processors, normality tests should always reject the null of normal distribution for large (though not insanely large) samples. And so, perversely, normality tests should only be used for small samples, when they presumably have lower power and less control over type I rate."

Is this a valid argument? Is this a well-known argument? Are there well known tests for a 'fuzzier' null hypothesis than normality?

-
For reference: I don't think that this needed to be community wiki. –  Shane Sep 8 '10 at 17:57
I wasn't sure there was a 'right answer'... –  shabbychef Sep 8 '10 at 18:01
–  Shane Sep 8 '10 at 18:03
In a certain sense, this is true of all test of a finite number of parameters. With $k$ fixed (the number of parameters on which the test is caried) and $n$ growthing without bounds, any difference between the two groups (no matter how small) will always break the null at some point. Actually, this is an argument in favor of bayesian tests. –  user603 Sep 8 '10 at 18:07
The thread at "Are large datasets inappropriate for hypothesis testing" discusses a generalization of this question. (stats.stackexchange.com/questions/2516/… ) –  whuber Sep 9 '10 at 20:17

It's not an argument. It is a (a bit strongly stated) fact that formal normality tests always reject on the huge sample sizes we work with today. It's even easy to prove that when n gets large, even the smallest deviation from perfect normality will lead to a significant result. And as every dataset has some degree of randomness...

Let me illustrate with the Shapiro-Wilks test. If you construct an almost-normal distribution and do a small simulation, you get more or less following results in R :

x <- replicate(100,{ # generates 100 different tests on each distribution
c(
shapiro.test(rnorm(10)+c(1,0,2,0,1))$p.value, shapiro.test(rnorm(100)+c(1,0,2,0,1))$p.value,
shapiro.test(rnorm(1000)+c(1,0,2,0,1))$p.value, shapiro.test(rnorm(5000)+c(1,0,2,0,1))$p.value
)
} # rnorm gives a random draw from the normal distribution
)
rownames(x)<-c("n10","n100","n1000","n5000")

rowMeans(x<0.05) # the proportion of significant deviations
n10  n100 n1000 n5000
0.04  0.04  0.20  0.87


so in 87% of the cases, the last distribution is not seen any more as a normal distribution. Yet, if you see the qq plots, you would never ever decide on a deviation from normality. Below you see as an example the qq-plots for one set of random samples

with p-values

 n10  n100 n1000 n5000
0.760 0.681 0.164 0.007

-
On a side note, the central limit theorem makes the formal normality check unnecessary in many cases when n is large. –  Joris Meys Sep 8 '10 at 23:19
yes, the real question is not whether the data are actually distributed normally but are they sufficiently normal for the underlying assumption of normality to be reasonable for the practical purpose of the analysis, and I would have thought the CLT based argument is normally [sic] sufficient for that. –  Dikran Marsupial Sep 9 '10 at 9:37
@joris-meys the central limit theorem does not help unless the population standard deviation is known. Very tiny disturbances in the random variable can distort the sample variance and make the distribution of a test statistic very far from the $t$ distribution, as shown by Rand Wilcox. –  Frank Harrell Aug 1 '13 at 11:42
This answer appears not to address the question: it merely demonstrates that the S-W test does not achieve its nominal confidence level, and so it identifies a flaw in that test (or at least in the R implementation of it). But that's all--it has no bearing on the scope of usefulness of normality testing in general. The initial assertion that normality tests always reject on large sample sizes is simply incorrect. –  whuber Oct 24 '13 at 21:16
Not one real life distribution is perfectly normal. So with large enough samples, all normality test should reject the null. So yes, SW does what it needs to do. But it is worthless for applied statistics. There's no point in going to eg a Wilcoxon when having a sample size of 5000 and an almost normal distribution. And that's what OP's remark was all about: does it make sense to test for normality when having large sample sizes? Answer: no. Why? because you detect (correctly) a deviation that doesn't matter for your analysis. As pointed out by the QQ plots –  Joris Meys Oct 25 '13 at 16:03

When thinking about whether normality testing is 'essentially useless', one first has to think about what it is supposed to be useful for. Many people (well... at least, many scientists) misunderstand the question the normality test answers.

The question normality tests answer: Is there convincing evidence of any deviation from the Gaussian ideal? With moderately large real data sets, the answer is almost always yes.

The question scientists often expect the normality test to answer: Do the data deviate enough from the Gaussian ideal to "forbid" use of a test that assumes a Gaussian distribution? Scientists often want the normality test to be the referee that decides when to abandon conventional (ANOVA, etc.) tests and instead analyze transformed data or use a rank-based nonparametric test or a resampling or bootstrap approach. For this purpose, normality tests are not very useful.

-
+1 for a good and informative answer. I find it useful to see a good explanation for a common misunderstanding (which I have incidentally been experiencing myself: stats.stackexchange.com/questions/7022/…). What I miss though, is an alternative solution to this common misunderstanding. I mean, if normality tests are the wrong way to go, how does one go about checking if a normal approximation is acceptable/justified? –  posdef Feb 10 '11 at 12:45
There's is not substitute for the (common) sense of the analyst (or, well, the researcher/scientist). And experience (learnt by trying and seeing: what conclusions do I get if I assume it is normal? What are the difference if not?). Graphics are your best friends. –  FairMiles Apr 5 '13 at 15:33
I like this paper, which makes the point you made: Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105(1), 156-166. –  Jeremy Miles Aug 20 at 20:18

I think that tests for normality can be useful as companions to graphical examinations. They have to be used in the right way, though. In my opinion, this means that many popular tests, such as the Shapiro-Wilk, Anderson-Darling and Jarque-Bera tests never should be used.

Before I explain my standpoint, let me make a few remarks:

• In an interesting recent paper Rochon et al. studied the impact of the Shapiro-Wilk test on the two-sample t-test. The two-step procedure of testing for normality before carrying out for instance a t-test is not without problems. Then again, neither is the two-step procedure of graphically investigating normality before carrying out a t-test. The difference is that the impact of the latter is much more difficult to investigate (as it would require a statistician to graphically investigate normality $100,000$ or so times...).
• It is useful to quantify non-normality, for instance by computing the sample skewness, even if you don't want to perform a formal test.
• Multivariate normality can be difficult to assess graphically and convergence to asymptotic distributions can be slow for multivariate statistics. Tests for normality are therefore more useful in a multivariate setting.
• Tests for normality are perhaps especially useful for practitioners who use statistics as a set of black-box methods. When normality is rejected, the practitioner should be alarmed and, rather than carrying out a standard procedure based on the assumption of normality, consider using a nonparametric procedure, applying a transformation or consulting a more experienced statistician.
• As has been pointed out by others, if $n$ is large enough, the CLT usually saves the day. However, what is "large enough" differs for different classes of distributions.

(In my definiton) a test for normality is directed directed against a class of alternatives if it is sensitive to alternatives from that class, but not sensitive to alternatives from other classes. Typical examples are tests that are directed towards skew or kurtotic alternatives. The simplest examples use the sample skewness and kurtosis as test statistics.

Directed tests of normality are arguably often preferable to omnibus tests (such as the Shapiro-Wilk and Jarque-Bera tests) since it is common that only some types of non-normality are of concern for a particular inferential procedure.

Let's consider Student's t-test as an example. Assume that we have an i.i.d. sample from a distribution with skewness $\gamma=\frac{E(X-\mu)^3}{\sigma^3}$ and (excess) kurtosis $\kappa=\frac{E(X-\mu)^4}{\sigma^4}-3.$ If $X$ is symmetric about its mean, $\gamma=0$. Both $\gamma$ and $\kappa$ are 0 for the normal distribution.

Under regularity assumptions, we obtain the following asymptotic expansion for the cdf of the test statistic $T_n$: $$P(T_n\leq x)=\Phi(x)+n^{-1/2}\frac{1}{6}\gamma(2x^2+1)\phi(x)-n^{-1}x\Big(\frac{1}{12}\kappa (x^2-3)-\frac{1}{18}\gamma^2(x^4+2x^2-3)-\frac{1}{4}(x^2+3)\Big)\phi(x)+o(n^{-1}),$$

where $\Phi(\cdot)$ is the cdf and $\phi(\cdot)$ is the pdf of the standard normal distribution.

$\gamma$ appears for the first time in the $n^{-1/2}$ term, whereas $\kappa$ appears in the $n^{-1}$ term. The asymptotic performance of $T_n$ is much more sensitive to deviations from normality in the form of skewness than in the form of kurtosis.

It can be verified using simulations that this is true for small $n$ as well. Thus Student's t-test is sensitive to skewness but relatively robust against heavy tails, and it is reasonable to use a test for normality that is directed towards skew alternatives before applying the t-test.

As a rule of thumb (not a law of nature), inference about means is sensitive to skewness and inference about variances is sensitive to kurtosis.

Using a directed test for normality has the benefit of getting higher power against ''dangerous'' alternatives and lower power against alternatives that are less ''dangerous'', meaning that we are less likely to reject normality because of deviations from normality that won't affect the performance of our inferential procedure. The non-normality is quantified in a way that is relevant to the problem at hand. This is not always easy to do graphically.

As $n$ gets larger, skewness and kurtosis become less important - and directed tests are likely to detect if these quantities deviate from 0 even by a small amount. In such cases, it seems reasonable to, for instance, test whether $|\gamma|\leq 1$ or (looking at the first term of the expansion above) $$|n^{-1/2}\frac{1}{6}\gamma(2z_{\alpha/2}^2+1)\phi(z_{\alpha/2})|\leq 0.01$$ rather than whether $\gamma=0$. This takes care of some of the problems that we otherwise face as $n$ gets larger.

-
Now this is a great answer! –  user603 Apr 4 at 10:45
Yea this should be the accepted, really fantastic answer –  noah Apr 14 at 19:24

IMHO normality tests are absolutely useless for the following reasons:

1. On small samples, there's a good chance that the true distribution of the population is substantially non-normal, but the normality test isn't powerful to pick it up.

2. On large samples, things like the T-test and ANOVA are pretty robust to non-normality.

3. The whole idea of a normally distributed population is just a convenient mathematical approximation anyhow. None of the quantities typically dealt with statistically could plausibly have distributions with a support of all real numbers. For example, people can't have a negative height. Something can't have negative mass or more mass than there is in the universe. Therefore, it's safe to say that nothing is exactly normally distributed in the real world.

-
Electrical potential difference is an example of a real-world quantity that can be negative. –  nico Sep 19 '10 at 13:03
@nico: Sure it can be negative, but there's some finite limit to it because there are only so many protons and electrons in the Universe. Of course this is irrelevant in practice, but that's my point. Nothing is exactly normally distributed (the model is wrong), but there are lots of things that are close enough (the model is useful). Basically, you already knew the model was wrong, and rejecting or not rejecting the null gives essentially no information about whether it's nonetheless useful. –  dsimcha Sep 22 '10 at 19:39
@dsimcha - I find that a really insightful, useful response. –  rolando2 May 4 '12 at 21:34
@dsimcha, the $t$-test and ANOVA are not robust to non-normality. See papers by Rand Wilcox. –  Frank Harrell Aug 1 '13 at 11:45

I think that pre-testing for normality (which includes informal assessments using graphics) misses the point.

1. Users of this approach assume that the normality assessment has in effect a power near 1.0.
2. Nonparametric tests such as the Wilcoxon, Spearman, and Kruskal-Wallis have efficiency of 0.95 if normality holds.
3. In view of 2. one can pre-specify the use of a nonparametric test if one even entertains the possibility that the data may not arise from a normal distribution.
4. Ordinal cumulative probability models (the proportional odds model being a member of this class) generalize standard nonparametric tests. Ordinal models are completely transformation-invariant with respect to $Y$, are robust, powerful, and allow estimation of quantiles and mean of $Y$.
-

Let me add one small thing:
Performing a normality test without taking its alpha-error into account heightens your overall probability of performing an alpha-error.

You shall never forget that each additional test does this as long as you don't control for alpha-error accumulation. Hence, another good reason to dismiss normality testing.

-
I presume you are referring to a situation where one first does a normality test, and then uses the result of that test to decide which test to perform next. –  Harvey Motulsky Sep 9 '10 at 16:07
I refer to the general utility of normality tests when used as method to determine whether or not it is appropriate to use a certain method. If you apply them in these cases, it is, in terms of probability of committing an alpha error, better to perform a more robust test to avoid the alpha error accumulation. –  Henrik Sep 10 '10 at 10:42
This does not make sense to me. Even if you decide between, say, an ANOVA or a rank-based method based on a test of normality (a bad idea of course), at the end of the day you would still only perform one test of the comparison of interest. If you reject normality erroneously, you still haven't reached a wrong conclusion regarding this particular comparison. You might be performing two tests but the only case in which you can conclude that factor such-and-such have an effect is when the second test also rejects $H_0$, not when only the first one does. Hence, no alpha-error accumulation… –  Gala Jun 8 '13 at 11:24
Another way a normality test could increase type I errors is if we're talking about "overall probability of performing an alpha-error." The test itself has an error rate, so overall, our probability of committing an error increases. Emphasis on one small thing too I suppose... –  Nick Stauner Nov 8 '13 at 15:49
@NickStauner That is exactly what I wanted to convey. Thanks for making this point even clearer. –  Henrik Nov 9 '13 at 12:25

Before asking whether a test or any sort of rough check for normality is "useful" you have to answer the question behind the question: "Why are you asking?"

For example, if you only want to put a confidence limit around the mean of a set of data, departures from normality may or not be important, depending on how much data you have and how big the departures are. However, departures from normality are apt to be crucial if you want to predict what the most extreme value will be in future observations or in the population you have sampled from.

-

The argument you gave is an opinion. I think that the importance of normality testing is to make sure that the data does not depart severely from the normal. I use it sometimes to decide between using a parametric versus a nonparametric test for my inference procedure. I think the test can be useful in moderate and large samples (when the central limit theorem does not come into play). I tend to use Wilk-Shapiro or Anderson-Darling tests but running SAS I get them all and they generally agree pretty well. On a different note I think that graphical procedures such as Q-Q plots work equally well. The advantage of a formal test is that it is objective. In small samples it is true that these goodness of fit tests have practically no power and that makes intuitive sense because a small sample from a normal distribution might by chance look rather non normal and that is accounted for in the test. Also high skewness and kurtosis that distinguish many non normal distributions from nomrla distribution are not easily seen in small samples.

-
While it certainly can be used that way, I don't think you will be more objective than with a QQ-Plot. The subjective part with the tests is when to decide that your data is to non-normal. With a large sample rejecting at p=0.05 might very well be excessive. –  Erik May 4 '12 at 17:56
Pre-testing (as suggested here) can invalidate the Type I error rate of the overall process; one should take into account the fact that a pre-test was done when interpreting the results of whichever test it selected. More generally, hypothesis tests should be kept for testing null hypothesis one actually cares about, i.e. that there is no association between variables. The null hypothesis that the data is exactly Normal doesn't fall into this category. –  guest May 4 '12 at 18:02
(+1) There is excellent advice here. Erik, the use of "objective" took me aback too, until I realized Michael's right: two people correctly conducting the same test on the same data will always get the same p-value, but they might interpret the same Q-Q plot differently. Guest: thank you for the cautionary note about Type I error. But why should we not care about the data distribution? Frequently that is interesting and valuable information. I at least want to know whether the data are consistent with the assumptions my tests are making about them! –  whuber May 4 '12 at 18:25
I strongly disagree. Both people get the same QQ-plot and same the p-value. To interpret the p-value you need to take into account the sample size and the violations of normality your test is particular sensitive to. So deciding what to do with your p-value is just as subjective. The reason you might prefer the p-value is that you believe the data could follow a perfect normal distribution - else it is just a question how quickly the p-value falls with sample size. Which is more, given a decent sample size the QQ-plot looks pretty much the same and remains stable with more samples. –  Erik May 4 '12 at 20:30
Erik, I agree that test results and graphics require interpretation. But the test result is a number and there won't be any dispute about it. The QQ plot, however, admits of multiple descriptions. Although each may objectively be correct, the choice of what to pay attention to is...a choice. That's what "subjective" means: the result depends on the analyst, not just the procedure itself. This is why, for instance, in settings as varied as control charts and government regulations where "objectivity" is important, criteria are based on numerical tests and never graphical results. –  whuber May 4 '12 at 21:54

I think a maximum entropy approach could be useful here. We can assign a normal distribution because we believe the data is "normally distributed" (whatever that means) or because we only expect to see deviations of about the same Magnitude. Also, because the normal distribution has just two sufficient statistics, it is insensitive to changes in the data which do not alter these quantities. So in a sense you can think of a normal distribution as an "average" over all possible distributions with the same first and second moments. this provides one reason why least squares should work as well as it does.

-

I think the first 2 questions have been thoroughly answered but I don't think question 3 was addressed. Many tests compare the empirical distribution to a known hypothesized distribution. The critical value for the Kolmogorov-Smirnov test is based on F being completely sppecified. It can be modified to test against a parametric distribution with parameters estimated. So if fuzzier means estimating more than two parameters then the answer to the question is yes. These tests can be applied the 3 parameter families or more. Some tests are designed to have better power when testing against a specific family of distributions. For example when testing normality the Anderson-Darling or the Shapiro-Wilk test have greater power than K-S or chi square when the null hypothesized distribution is normal. Lillefors devised a test that is preferred for exponential distributions.

-

Tests where "something" important to the analysis is supported by high p-values are I think wrong headed. As others pointed out, for large data sets, a p-value below 0.05 is assured. So, the test essentially "rewards" for small and fuzzy data sets and "rewards" for a lack of evidence. Something like qq plots are much more useful. The desire for hard numbers to decide things like this always (yes/no normal/not normal) misses that modeling is partially an art and how hypotheses are actually supported.

-
samples are generally small in the sense of big populations that may be frequently dispersed. Moreover, we must ensure a correct identification of target population. Even a small sample is good if we are getting data through properly conducted controlled experiments. –  subhash c. davar Dec 14 '13 at 3:55
It remains that a large sample that is nearly normal will have a low p-value while a smaller sample that is not nearly as normal will often not. I do not think that large p-values are useful. Again, they reward for a lack of evidence. I can have a sample with several million data points, and it will nearly always reject the normality assumption under these tests while a smaller sample will not. Therefore, I find them not useful. If my thinking is flawed please show it using some deductive reasoning on this point. –  wvguy8258 Jul 9 at 7:43

One good use of normality test that I don't think has been mentioned is to determine whether using z-scores is okay. Let's say you selected a random sample from a population, and you wish to find the probability of selecting one random individual from the population and get a value of 80 or higher. This can be done only if the distribution is normal, because to use z-scores, the assumption is that the population distribution is normal.

But then I guess I can see this being arguable too...

-
Value of what? Mean, sum, variance, an individual observation? Only the last one relies on the assumed normality of the distribution. –  whuber Sep 29 '13 at 16:12
i meant individual –  Hotaka Sep 29 '13 at 16:29
Thanks. Your answer remains so vague, though, that it is difficult to tell what procedures you are referring to and impossible to assess whether your conclusions are valid. –  whuber Sep 29 '13 at 16:33
The problem with this use is the same as with other uses: The test will be dependent on sample size, so, it's essentially useless. It doesn't tell you whether you can use z scores. –  Peter Flom May 31 at 0:24