Intepretation of $p$-values for a Kolmogorov-Smirnov test

Question

I'm using a Kolmogorov-Smirnov test to examine how normal (or Gaussian) some data sets I have are. I am having trouble with interpreting the $p$-values (surprise surprise).

My present understanding is that if the returned $p$-value is $p > 0.05$ this suggests that the data is normally distributed, while datasets which produce a Kolmogorov-Smirnov test result $p < 0.05$ suggest that the data is not normally distributed.

Firstly, is my present understanding/interpretation correct?

Secondly I performed some very basic simulations. I simply generate data sets with $1000$ points which are normally distributed and perform a Kolmogorov-Smirnov test on each one. If I look at the distribution of $p$-values extracted from this, I see a square distribution with values ranging between 0 and 1.

I don't understand this -- I guess I don't really understand $p$-values as I expected for data which definitely is normally distributed (I generated it to be such) that there would be a median $p$-value.

Answer 1

You're on the right track. Two facts are key for understanding what's going on.

First, the null hypothesis $(H_\emptyset$) of the one-sample K-S test is that the sample data comes from the "reference" distribution. When it is being used as a normality test, that reference distribution is a Gaussian.

Second, for any test, p-values are uniformly distributed between 0 and 1 under the null hypothesis, assuming the test's assumptions are met. See this question for more details. While you should expect small $p$-values if the alternate hypothesis were true, it is not the case that you should expect only large $p$-values under the null hypothesis.

Combining these two facts accounts for your simulation: you set up conditions where the null hypothesis is definitely true, and the resulting p-values form a "square" histogram where each bin has approximately the same number of values. If you were to draw samples from a different distribution (e.g., an exponential distribution), you would instead find $p$-values clustered near 0, since it is unlikely that those values could have come from a Gaussian (your $H_\emptyset$).