4
$\begingroup$

My shortened data is:

y <- c (2,2,1,5,6,7,1,2,1,6,6,7,3,2,4,4,4,4,3,3,9,1,1,9)

I firstly normalize my data:

y_scale <- scale(y)

Then, I generate a model dataset with normal distribution based on y_scale's mean and stdev:

y_norm <- rnorm(n=24, m=mean(y_scale), sd=sd(y_scale))

To check if my data fits the normal distribution, I do

ks.test(y_scale,y_norm)

I found the result is as follows:

Two-sample Kolmogorov-Smirnov test

data:  y_scale and y_norm 
D = 0.2083, p-value = 0.6749
alternative hypothesis: two-sided 

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

Here, my question is:

(1) My real data set has ~ 700,000 numbers, I found I cannot use shapiro.test.

shapiro.test(y_scale)
Error in shapiro.test(y_scale) : sample size must be between 3 and 5000

(2) Is the p-value calculated above by ks.test wrong? How to solve this problem of p-values? 

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

(3) The reasons why I tried to use ks.test instead of other methods, is because I want to compare with other model datasets that have other distribution functions. It seems to me that I can simply replace y_norm with other model dataset, and compare their p-values or D-values (the smaller the better), to choose which distribution function fits my data most.

(4) Is it a must to normalize my data first?

Improve this question
$\endgroup$
3
  • 5
    $\begingroup$ Unless your data are artificially generated by a particular, known distribution, I can guarantee that any reasonably effective distribution test will reject them! Nevertheless, it can be meaningful to use a goodness of fit statistic, like the KS statistic, to measure the deviation between the distribution of your data and any ideal distribution (there will be no problem with ties or discreteness, either), but the p-values are worthless. $\endgroup$
    – whuber ♦
    Commented Jun 22, 2011 at 16:31
  • 1
    $\begingroup$ Probably QQ-plots might guide you also, though I'm not sure what is the reason for scaling at such sample size if you do know that the data has two finite moments then scaling will bring you to the central limit theorem conclusions. $\endgroup$
    – Dmitrij Celov
    Commented Jun 22, 2011 at 18:43
  • $\begingroup$ Interesting read: stats.stackexchange.com/questions/2492/… $\endgroup$
    – nico
    Commented Jun 22, 2011 at 20:23

3 Answers 3

Reset to default
5
$\begingroup$

Why are you testing for normality?

As stated by @user603 the discreteness of your data suggests that it is not normal. But are you interested in normal enough?

With 700,000 data points you will have power to detect differences from normality that are very minor and probably not important (I just compared a sample from a t distribution with 100 df and got a p-value of 0.011). If you are doing the normality test in preparation to using normal theory inference (t-tests, confidence intervals, etc.) then for most population distributions 700,000 should be enough for the central limit theorem to let you use the normal theory inference, but knowledge about the population is probably more important here than any test of normality.

Also, if your data really is discrete, then normalizing ruins the discreteness and could be misleading. For meaningful results you should not normalize.

Computing the mean and standard deviation after normalizing is meaningless since you have set the mean to 0 and standard deviation to 1.

The KS test is designed to test against a fully specified distribution, normalizing the data first is equivalent to comparing to a distribution with estimated parameters which will also invalidate the p-value.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have some questions. Isn't the data always discrete? We can depict numbers only up to a number of digits. Can't the marks in a school be normal distributed? $\endgroup$
    – Bogdan Lataianu
    Commented Jun 23, 2011 at 7:00
  • $\begingroup$ There are 2 concepts here, the actual data entered into the computer which will always be discrete, and the theoretical mechanism for generating the data which we can theorize about being continous (even though the data we enter will be a discrete approximation to this). Your data is all integers which means that either the mechanism for generating the data is discrete, like counts from a poisson or binomial where it is impossible to get fractional counts, or you have done a lot of rounding. This is one of the reasons that understanding beats testing. $\endgroup$
    – Greg Snow
    Commented Jun 23, 2011 at 19:17
2
$\begingroup$
  1. If your data is discrete, it cannot possible be drawn from a continuous distribution.
  2. Try ks.test(y,"pnorm",mean(y),sd(y)).
Improve this answer
$\endgroup$
4
  • $\begingroup$ I tried to do this as follows, but I don't know if I have misunderstood your reply. I found the D value are different as show below. However, the p-value and the warning message is the same as below. Could you mind to give me some more guidance? $\endgroup$
    – evdstat
    Commented Jun 22, 2011 at 16:19
  • $\begingroup$ One-sample Kolmogorov-Smirnov test data: y D = 0.0703, p-value < 2.2e-16 alternative hypothesis: two-sided Warning message: In ks.test(y, "pnorm", mean(y), sd(y)) : cannot compute correct p-values with ties $\endgroup$
    – evdstat
    Commented Jun 22, 2011 at 16:20
  • $\begingroup$ > ks.test(y, y_norm) Two-sample Kolmogorov-Smirnov test data: y and y_norm D = 0.9386, p-value < 2.2e-16 alternative hypothesis: two-sided Warning message: In ks.test(y, y_norm) : cannot compute correct p-values with ties > $\endgroup$
    – evdstat
    Commented Jun 22, 2011 at 16:21
  • $\begingroup$ i can't help but point out again that if your data is discrete (as it seems to be) it cannot possible be a draw from a Gaussian (i.e. the correct p-value in this case is 0). $\endgroup$
    – user603
    Commented Jun 22, 2011 at 16:54
2
$\begingroup$

As suggested by user603, you can test directly by comparing with the normal distribution.
As you can see at help page the $p$-values are not exact if there are ties because "the continuous distributions do not generate them." So the theory (and hence the algorithm) does not allow ties.
I did some testings using
x <- rnorm(N)
ks.test(x,"pnorm")
and increased the size of $N$. I noticed the $p$-value is unreliable, jumping up and down randomly. And there were no ties (give credit to R for that). However, the $D$-value is consistent and decreases with the increase of the size. So I suggest you to use the $D$-value.
And about the normalizing thing, there is no such requirement. So I see no point in doing it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.