I'm using a Kolmogorov-Smirnov test to examine how normal (or Gaussian) some data sets I have are. I am having trouble with interpreting the $p$-values (surprise surprise).

My present understanding is that if the returned $p$-value is $p > 0.05$ this suggests that the data is normally distributed, while datasets which produce a Kolmogorov-Smirnov test result $p < 0.05$ suggest that the data is not normally distributed.

Firstly, is my present understanding/interpretation correct?

Secondly I performed some very basic simulations. I simply generate data sets with $1000$ points which are normally distributed and perform a Kolmogorov-Smirnov test on each one. If I look at the distribution of $p$-values extracted from this, I see a square distribution with values ranging between 0 and 1.

I don't understand this -- I guess I don't really understand $p$-values as I expected for data which definitely is normally distributed (I generated it to be such) that there would be a median $p$-value.


You're on the right track. Two facts are key for understanding what's going on.

First, the null hypothesis $(H_\emptyset$) of the one-sample K-S test is that the sample data comes from the "reference" distribution. When it is being used as a normality test, that reference distribution is a Gaussian.

Second, for any test, p-values are uniformly distributed between 0 and 1 under the null hypothesis, assuming the test's assumptions are met. See this question for more details. While you should expect small $p$-values if the alternate hypothesis were true, it is not the case that you should expect only large $p$-values under the null hypothesis.

Combining these two facts accounts for your simulation: you set up conditions where the null hypothesis is definitely true, and the resulting p-values form a "square" histogram where each bin has approximately the same number of values. If you were to draw samples from a different distribution (e.g., an exponential distribution), you would instead find $p$-values clustered near 0, since it is unlikely that those values could have come from a Gaussian (your $H_\emptyset$).

| cite | improve this answer | |
  • $\begingroup$ I think I understand a bit more now. So while I observe a square or uniform distribution, I can say 95% of the data sets satisfy the the K-S tests? And I can safely say there is only a 5% chance a truly normal data set will be rejected? $\endgroup$ – Q.P. Oct 7 at 14:59
  • $\begingroup$ @Q.P. What do you mean that 95% of data sets will satisfy the KS test? $\endgroup$ – Dave Oct 7 at 16:56
  • $\begingroup$ @Dave I'll illustrate what I mean by adding my simulation and the results. $\endgroup$ – Q.P. Oct 7 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.