7
$\begingroup$

I have a large dataset (500000 data, V1 column include all the data).

x <- read.csv("mydata.csv", header=F)
hist(x)

Which gives:

Histogram

Looking at the data, I believe it is not a normal distribution. As a further check, I constructed a qqplot:

x_norm <- (x$V1 - mean(x$V1))/sd(x$V1)
qqnorm(x_norm); abline(0, 1)

which gave:

QQ-plot

To check the goodness of fit of x$V1 (rawdata) to a normal distribution, I used:

rnorm <- rnorm(500000, mean(x$V1), sd(x$V1))
cc <- cbind(rnorm, x$V1)
g <- goodfit(cc, method="MinChisq")
summary(g)

         Goodness-of-fit test for poisson distribution

             X^2 df      P(> X^2)
Pearson 914.5227 17 1.679266e-183
Warning message:
In summary.goodfit(g) : Chi-squared approximation may be incorrect

With plot(g) giving:

Does this seem correct? Can I confidently conclude my dataset X$V1 is or is not a normal distribution?

Based on the above analysis, what other distribution should I test?

$\endgroup$
6
$\begingroup$

I would not rely on p-values for any test of normality (or for much else, frankly). Look at the graphs.

You can, a priori, say that EVERY distribution is non-normal. If you have a large dataset the nonnormality will be statistically significant. The questions are HOW non-normal? Non-normal in what ways? and What are the consequences?

None of these questions is answered by any test of normality or statistical significance.

Why are you testing normality? If it's a test of residuals for some linear model, there was a great quote from George Box ... something like this is "like sending out a rowboat to see if the water is calm enough for an ocean liner"

$\endgroup$
5
$\begingroup$

All I can say is that your eyes are one of your better EDA tools. If your data (with 500,000 observations) doesn't look normal, then there's no reason to even perform a statistical test for normality. Especially with that many data points, any slight deviation from normality should make you fail the test.

It looks like your data is actually discrete, too. You should consider fitting a Binomial or Poisson or some other discrete distribution to the data.

$\endgroup$
  • $\begingroup$ Thanks Max. Could you mind to let me know where can i find how does other distribution should look like? As I have not much experience that I cannot tell which population does it look like when I look at the population chart 1. $\endgroup$ – evdstat Jun 23 '11 at 7:06
  • $\begingroup$ I also want to ask for your comments, when I look at the plot(g) results, (i.stack.imgur.com/E0BTd.png), it seems to me that they are quite good, when I look at the P value (1.679266e-183) above, it seems to be quite small. Could you mind to teach me how to decide whether my data fit the normal distribution in Chi-square test? $\endgroup$ – evdstat Jun 23 '11 at 7:09
  • 2
    $\begingroup$ I probably shouldn't have told you your data looks discrete without actually seeing some of the numbers. Would you mind posting a summary/example of the data you're working with? $\endgroup$ – assumednormal Jun 23 '11 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.