7
$\begingroup$

I have a 1D data set with 83163 data points, and I would like to know whether the data follows a normal distribution.

I tried using shapiro.test and ks.test in R:

d is a vector containing the data

shapiro.test(sample(d, 5000))

    Shapiro-Wilk normality test

data:  sample(d, 5000) 
W = 0.9694, p-value < 2.2e-16

(Repeated several times. Note subsampling.)

ks.test(d, dnorm, mean=mean(d), sd=sd(d))

    One-sample Kolmogorov-Smirnov test

data:  d 
D = 1, p-value < 2.2e-16
alternative hypothesis: two-sided 

Warning message:
In ks.test(d, dnorm, mean = mean(d), sd = sd(d)) :
  cannot compute correct p-values with ties

Both tests indicate that the data distribution is not normal.

So I tried plotting the data (black), and it appears to be "taller" than a normal distribution with mean and sd estimated from the data (blue).

http://dl.dropbox.com/u/38050492/data-distrib.png

I wondered if the variance is over-estimated due to outliers, so I tried calculating Winsorized variance. I heuristically matched the peak to the data distribution, but I cannot get a good fit (red).

Edit:

qqplot also suggest non-normality.

http://dl.dropbox.com/u/38050492/qqplot.png

Is there a distribution that may better model the data?

The reason why I wanted to check normality is because others have done two-sample z-tests, and modeled the data using Gaussian distributions.

To make the long story short, the math work out much nicely if the data distribution is assumed to be normal. As the normality assumption goes beyond simple application of parametric tests, I don't know how robust the results are when the data is not normally distributed...

There does appear to a considerable deviation from the normality in terms of the kurtosis of the distribution. And this deviation is consistent from dataset to dataset...

Improve this question
$\endgroup$
8
  • $\begingroup$ See also here: stats.stackexchange.com/questions/16611/… $\endgroup$
    – xmjx
    Commented Oct 19, 2011 at 22:35
  • $\begingroup$ But I cannot fit a normal distribution to the data using the estimated mean and variance, or even an Winsorized variance. Is the data really truly normal? Why does it appear 'taller' given the estimated variance? In the shapiro test, I am only use a sub-sample of the data. If the problem is truly due to large sample size, should sub-sampling not solve the problem? $\endgroup$
    – David Shih
    Commented Oct 19, 2011 at 22:37
  • $\begingroup$ Some of the answers in the linked question have some good advice. Personally, I'd like to see a qq-plot. What you're going to do with the data afterwards also determines how much you really care about the 'significantly non-normal' result. $\endgroup$
    – user5594
    Commented Oct 19, 2011 at 22:51
  • $\begingroup$ qqplot seems to suggest non-normality (see updated post)... $\endgroup$
    – David Shih
    Commented Oct 19, 2011 at 23:13
  • $\begingroup$ Karl beat me to do it. That qq-plot is nasty. $\endgroup$
    – user5594
    Commented Oct 19, 2011 at 23:49

1 Answer 1

Reset to default
12
$\begingroup$

It's definitely not normal, and it's not just the large sample size. That qq-plot is really clear. The wikipedia page about kurtosis may be useful to you; it mentions the Pearson type VII family of distributions, with an image of densities quite similar to yours.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.