12
$\begingroup$

The Kolgomorov-Smirnov test, Shapiro test, etc.... all reject the hypothesis that a distribution is normal. Yet when I plot the normal quantiles and and histogram, the data is clearly normal. Maybe because the power of the tests are high?

The sample size is around 650. So shouldn't at least one of these tests fail to reject the null hypothesis?

Results:

           Kolmogorov-Smirnov    D          0.05031          Pr > D       <0.010
           Cramer-von Mises      W-Sq       0.30003          Pr > W-Sq    <0.005
           Anderson-Darling      A-Sq       1.66965          Pr > A-Sq    <0.005
           Chi-Square            Chi-Sq  3250.43596     18   Pr > Chi-Sq  <0.001
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Welcome to the site. The power may indeed be an issue. Can you post your results, so that we could be more specific? $\endgroup$
    – StasK
    Oct 6, 2011 at 17:47
  • 1
    $\begingroup$ It's almost impossible to assess normality with much accuracy by looking at a histogram or the quantiles. The first three of these tests measure deviations in a probability plot (Normal q-q plot), so how linear does that plot look? $\endgroup$
    – whuber ♦
    Oct 6, 2011 at 20:14

5 Answers 5

Reset to default
14
$\begingroup$

Normality testing is a waste of time and your example illustrates why. With small samples, the normality test has low power, so decisions about what statistical models to use need to be based on a priori knowledge. In these cases failure to reject the null doesn't prove that the null is even approximately true at the population level.

When you have large samples, normality tests become ridiculously powerful, but they don't tell you anything you didn't already know. No real quantity is exactly normally distributed. The normal distribution is just a mathematical abstraction that's a good enough approximation in a lot of cases. The simplest proof of this is that there is no real quantity (at least none that I can think of) that could take any real number as its value. For example, there are only so many molecules in the universe. There are only so many dollars in the money supply. The speed of light is finite. Computers can only store numbers of a finite size, so even if something did have a support of all real numbers, you wouldn't be able to measure it.

The point is that you already knew your data wasn't exactly normally distributed but the normality tests tell you nothing about how non-normal the data is. They give you absolutely no hint as to whether your data is approximately normally distributed such that statistical inference methods that assume normality would give correct answers. Ironically, common tests (e.g. the T-test and ANOVA) that assume normality are more robust to non-normality at large sample sizes.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Following on from your answer, I posted a question about what constitutes a good index of non-normality: stats.stackexchange.com/questions/16646/… Any thoughts? $\endgroup$
    – Jeromy Anglim
    Oct 7, 2011 at 6:12
  • $\begingroup$ Regarding everthing in the world being quantized: can't discrete data be normally distributed, too? $\endgroup$
    – xmjx
    Oct 7, 2011 at 11:04
  • $\begingroup$ One more comment on the computer issue: please note that the mechanism often used to store decimal numbers in computers has a different granularity for the range of small numbers and the large numbers. So the minimum difference between to numbers that the computer is able to store is smaller for small numbers and larger for large numbers. For a computer, 100000.1 and 100000.2 might be the same while 0.1 and 0.2 are not. (Just an example - in the real world it's not that bad.) $\endgroup$
    – xmjx
    Oct 7, 2011 at 11:16
  • $\begingroup$ @xmjx: Discrete data can be approximately normally distributed, meaning that it's close enough for almost any practical purpose. However, in theory any discrete distribution will fail some tests for normality if the sample size is large enough. The normal distribution is continuous and there's no way around that. $\endgroup$
    – dsimcha
    Oct 7, 2011 at 17:08
  • $\begingroup$ @dsimcha But the normal distribution is just a probability density function which could predict the number of observations in a given bin of the discrete variable. So, I would understand if you said "no real variable is exactly normally distributed and this is why normality tests will fail at some point". But for "discrete data cannot be normally distributed since it's not continuous" I'd like some reference. I'm really interested in that kind of stuff. Not wanting to start a fight here. $\endgroup$
    – xmjx
    Oct 8, 2011 at 9:10
4
$\begingroup$

This doesn't surprise me --- with a large enough sample size, any good test should reject the null hypothesis, unless the data generating distribution is truly (and exactly) normal.

With hypothesis testing, one is usually interested in finding a "powerful" test, which is a test that can find very small deviations from the null hypothesis, with as little data as possible.

Try running the test with a subsample of size 20, 50, 100, 200, and see at what size the tests start rejecting. It's easy to see whether a histogram is symmetric and generally bell-shaped, but the tails of the distribution are harder to assess by eye. Perhaps there are outliers in the data that are causing the tests to reject? If there are, see what happens when you prune them out.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the answer. The purpose is testing whether the residuals are normal. I guess looking at a normal-quantile plot and seeing whether it lies on y =x is the best bet? $\endgroup$
    – Robbie
    Oct 6, 2011 at 18:00
  • $\begingroup$ @Robbie If you're just interested in trying to figure out if your residuals are normal, a visual inspection should be fine. Statistical testing of the distribution isn't really needed for that - as has been noted, it's going to pick up any deviation from normality, even one that doesn't really matter. $\endgroup$
    – Fomite
    Oct 6, 2011 at 18:04
  • $\begingroup$ @EpiGrad I disagree. Tests for normality have notoriously low power. See my answer above. Edit on the other hand, regression is pretty robust to non-normality, so I'd agree that if it looks normal, you're probably fine for that purpose. $\endgroup$
    – David J. Harris
    Oct 6, 2011 at 18:23
  • $\begingroup$ @David J. Harris: "Notoriously low power"? For sample sizes of 650? This is contrary to everything I have read or experienced. Do you have a citation? $\endgroup$
    – whuber ♦
    Oct 6, 2011 at 18:54
  • $\begingroup$ @DavidJ.Harris I think at the core, low power or spurious significance due to a large sample, the entire exercise is unnecessary for routine examination of the normality assumption. $\endgroup$
    – Fomite
    Oct 6, 2011 at 20:05
3
$\begingroup$

The probable cause is that your data is very slightly non-normal and your sample size is big enough to reveal this.

If the distribution truly is normal then it should usually pass these tests, as in the following R example where all but one of the tests are passed.

> require(nortest)
> 
> set.seed(1)
> dat <- rnorm(650,mean=100, sd=5)
> 
> ad.test(dat)

        Anderson-Darling normality test

data:  dat 
A = 0.439, p-value = 0.2924

> cvm.test(dat)

        Cramer-von Mises normality test

data:  dat 
W = 0.0882, p-value = 0.1619

> lillie.test(dat)

        Lilliefors (Kolmogorov-Smirnov) normality test

data:  dat 
D = 0.0334, p-value = 0.08196

> pearson.test(dat)

        Pearson chi-square normality test

data:  dat 
P = 37.96, p-value = 0.035

> sf.test(dat)

        Shapiro-Francia normality test

data:  dat 
W = 0.9978, p-value = 0.5186

> shapiro.test(dat)

        Shapiro-Wilk normality test

data:  dat 
W = 0.9981, p-value = 0.675

You might want to do a qqplot and if this is close enough to a straight line then you may decide to treat it as being close enough to normality for your purposes. It rather depends on what those purposes are.

Improve this answer
$\endgroup$
5
  • $\begingroup$ The purposes are for testing to see whether the residuals are normal in linear regression. $\endgroup$
    – Robbie
    Oct 6, 2011 at 17:58
  • 1
    $\begingroup$ @Robbie. Apparently they are not, but they may be close enough for this not to matter much. Try the qqplot. $\endgroup$
    – Henry
    Oct 6, 2011 at 18:10
  • $\begingroup$ The Pearson chi square result looks like the data is not normally distributed. Just saying. What to do with such a result? $\endgroup$
    – xmjx
    Oct 7, 2011 at 11:05
  • $\begingroup$ @xmjx: Not much - If you apply a 0.05 criterion, then you should not be suprised if you get a false positive 5% of the time. $\endgroup$
    – Henry
    Oct 7, 2011 at 16:33
  • $\begingroup$ @Henry I know. What I mean: choosing any normality test upfront has some probability of choosing one that will say "significant". So is it better to run a battery and then ... what? Average? Go with the majority vote? $\endgroup$
    – xmjx
    Oct 8, 2011 at 9:01
2
$\begingroup$

Let me disagree with the answer from dsimcha: "Normality testing is a waste of time and your example illustrates why.". Normality testing is never a waste of time, you can always learn from your data. Furthermore, there are some condition you must test before performing some analysis (i.e. ANOVA, regression, etc.). Relative large sample sizes are better to be tested with plot (QQplot, histogram). In such cases, visualization give much more information about multimodal behavior and so on.

ANOVA and regression are robust to non-normality when dealing large sample sizes but the main type of data that cause problems is multimodal data samples.

With small sample size the Kolgomorov-Smirnov test is the best option mainly due to its sensitivity.

Improve this answer
$\endgroup$
1
$\begingroup$

I'm going to disagree slightly with the other answers posted so far: these tests for normality have notoriously little power, even with relatively large sample sizes, at least for certain kinds of deviations.

Here's a quick example. I generated a mixture of two normals whose means are separated by a whole sd.

set.seed(1)
reps = replicate(
  10000, 
  shapiro.test(c(rnorm(325, mean = 0), rnorm(325, mean = 1)))$p.value
)
mean(reps < .05)
[1] 0.0525

Considering that it would "detect" deviations from normality 5% of the time even if it were truly normal, that's not very impressive.

Here's another example: I add uniform noise across a range the size of two standard deviations. This one is quite visibly non-normal.

set.seed(1)
reps = replicate(
  10000, 
  shapiro.test(rnorm(650) + 2 * runif(650))$p.value
)
mean(reps < .05)
[1] 0.0523

Again, extremely low power for a pretty big departure from normality.

Are you sure you're reading the qqplot correctly? Could you upload it so we could see it?

Edit on the other hand, regression is fairly robust to non-normality, so I'd agree that visual inspection is likely to be enough for most purposes.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ I think you may be confusing the apparent size of a "deviation," as written in a formula, with the actual deviation between two distributions. In both cases the result is remarkably close to Normal. In case 1, the PDF is difficult to distinguish visually from a Normal PDF; all its odd moments are (obviously) $0$; its kurtosis is $73/25$, just a tiny bit less than that of a standard Normal ($75/25$), etc. The fact that the Shapiro-Wilks test has any power to identify this mixture as non-Normal with a pseudo sample of 625 is remarkable. The second case is similar. $\endgroup$
    – whuber ♦
    Oct 6, 2011 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.