Intepretation of $p$-values for a Kolmogorov-Smirnov test

I'm using a Kolmogorov-Smirnov test to examine how normal (or Gaussian) some data sets I have are. I am having trouble with interpreting the $$p$$-values (surprise surprise).

My present understanding is that if the returned $$p$$-value is $$p > 0.05$$ this suggests that the data is normally distributed, while datasets which produce a Kolmogorov-Smirnov test result $$p < 0.05$$ suggest that the data is not normally distributed.

Firstly, is my present understanding/interpretation correct?

Secondly I performed some very basic simulations. I simply generate data sets with $$1000$$ points which are normally distributed and perform a Kolmogorov-Smirnov test on each one. If I look at the distribution of $$p$$-values extracted from this, I see a square distribution with values ranging between 0 and 1.

I don't understand this -- I guess I don't really understand $$p$$-values as I expected for data which definitely is normally distributed (I generated it to be such) that there would be a median $$p$$-value.

You're on the right track. Two facts are key for understanding what's going on.

First, the null hypothesis $$(H_\emptyset$$) of the one-sample K-S test is that the sample data comes from the "reference" distribution. When it is being used as a normality test, that reference distribution is a Gaussian.

Second, for any test, p-values are uniformly distributed between 0 and 1 under the null hypothesis, assuming the test's assumptions are met. See this question for more details. While you should expect small $$p$$-values if the alternate hypothesis were true, it is not the case that you should expect only large $$p$$-values under the null hypothesis.

Combining these two facts accounts for your simulation: you set up conditions where the null hypothesis is definitely true, and the resulting p-values form a "square" histogram where each bin has approximately the same number of values. If you were to draw samples from a different distribution (e.g., an exponential distribution), you would instead find $$p$$-values clustered near 0, since it is unlikely that those values could have come from a Gaussian (your $$H_\emptyset$$).

• I think I understand a bit more now. So while I observe a square or uniform distribution, I can say 95% of the data sets satisfy the the K-S tests? And I can safely say there is only a 5% chance a truly normal data set will be rejected? Oct 7, 2020 at 14:59
• @Q.P. What do you mean that 95% of data sets will satisfy the KS test?
– Dave
Oct 7, 2020 at 16:56
• @Dave I'll illustrate what I mean by adding my simulation and the results. Oct 7, 2020 at 17:13