# Is my data distribution normal? (Tried Shapiro and Kolmogornov-Smirnov tests)

I have a 1D data set with 83163 data points, and I would like to know whether the data follows a normal distribution.

I tried using shapiro.test and ks.test in R:

d is a vector containing the data

shapiro.test(sample(d, 5000))

    Shapiro-Wilk normality test

data:  sample(d, 5000)
W = 0.9694, p-value < 2.2e-16


(Repeated several times. Note subsampling.)

ks.test(d, dnorm, mean=mean(d), sd=sd(d))

    One-sample Kolmogorov-Smirnov test

data:  d
D = 1, p-value < 2.2e-16
alternative hypothesis: two-sided

Warning message:
In ks.test(d, dnorm, mean = mean(d), sd = sd(d)) :
cannot compute correct p-values with ties


Both tests indicate that the data distribution is not normal.

So I tried plotting the data (black), and it appears to be "taller" than a normal distribution with mean and sd estimated from the data (blue). I wondered if the variance is over-estimated due to outliers, so I tried calculating Winsorized variance. I heuristically matched the peak to the data distribution, but I cannot get a good fit (red).

Edit:

qqplot also suggest non-normality. Is there a distribution that may better model the data?

The reason why I wanted to check normality is because others have done two-sample z-tests, and modeled the data using Gaussian distributions.

To make the long story short, the math work out much nicely if the data distribution is assumed to be normal. As the normality assumption goes beyond simple application of parametric tests, I don't know how robust the results are when the data is not normally distributed...

There does appear to a considerable deviation from the normality in terms of the kurtosis of the distribution. And this deviation is consistent from dataset to dataset...

• See also here: stats.stackexchange.com/questions/16611/… – xmjx Oct 19 '11 at 22:35
• But I cannot fit a normal distribution to the data using the estimated mean and variance, or even an Winsorized variance. Is the data really truly normal? Why does it appear 'taller' given the estimated variance? In the shapiro test, I am only use a sub-sample of the data. If the problem is truly due to large sample size, should sub-sampling not solve the problem? – David Shih Oct 19 '11 at 22:37
• Some of the answers in the linked question have some good advice. Personally, I'd like to see a qq-plot. What you're going to do with the data afterwards also determines how much you really care about the 'significantly non-normal' result. – user5594 Oct 19 '11 at 22:51
• qqplot seems to suggest non-normality (see updated post)... – David Shih Oct 19 '11 at 23:13
• Karl beat me to do it. That qq-plot is nasty. – user5594 Oct 19 '11 at 23:49