# Non-uniform distribution of p-values

I'm running a Monte Carlo simulation to check whether a distribution generated by my model fits a pre-defined target distribution where the parameters of the target distribution are estimated from the data.

I ran a KS-test on the output of my simulation against the target distribution and checked p-values, and they were consistently high (p~1), suggesting that the fit was indeed good.

Just as a sense check I generated ~10^3 samples to check the distribution of these p-values. From my understanding under the null hypothesis of identical disributions these p-values should be uniformly distributed.

However, oddly, the distribution of p-values is skewed towards 1 (see distribution of p-values below) - it seems an overly large number of the samples fit the target distribution closely.

I'm struggling to understand how to interpret this. Obviously if it was skewed towards 0 it's pretty clear that the samples don't fit the target. How should one interpret the inverse situation as described?

Edit: Added that parameters are estimated during the fitting process. This appears to explain the skew.

• you may want to share us the simulation code? – TPArrow Jul 2 '19 at 9:39
• Is your target distribution truly fully specified? Or are some of its parameters estimated? – Stephan Kolassa Jul 2 '19 at 9:45
• This is characteristic of a misapplication of the K-S test where you are comparing data to a distribution whose parameters have been estimated from the same data. However, since you don't provide any specifics about how you are performing the test, we haven't sufficient information even to know whether that's the cause of the problem. – whuber Jul 2 '19 at 13:42
• @whuber I added an edit stating that parameters have been estimated. Thank you - this does actually answer my question as I didn't understand what I had mistaken here. In this case I take it best practice would be to sample the true (fitted) distribution to uncover the distribution of the KS statistic under the null? – Sue Doh Nimh Jul 2 '19 at 14:08
• That's certainly a valid approach. Tests have been devised to account for fitting the parameters in special cases: see, for instance, the Lilliefors test. Be very careful of Web examples of of applying the K-S test, because this is a trap into which many have fallen and (unlike you) not all have become aware that something is remiss. – whuber Jul 2 '19 at 14:13

As whuber has commented: the Kolmogorov-Smirnov test is only valid as a comparison against a fully specified distribution. You cannot use it to compare an observed distribution against a distribution whose parameters have been estimated based on your observed sample. If you do so, your p-values will not be uniformly distributed under the null hypothesis, but show the exact pattern you have observed.

This is unfortunately an extremely common error, which you can very often find in online tutorials.

As a little illustration, let us simulate $$x_1, \dots, x_{20}\sim N(0,1)$$, then run a K-S test first against a fully specified $$N(0,1)$$ distribution, then against an estimated $$N(\hat{\mu},\hat{\sigma}^2)$$ distribution, where $$\hat{\mu}$$ and $$\hat{\sigma}^2$$ are estimated based on $$x_1, \dots, x_{20}$$. Record the $$p$$ value. Do this 10,000 times. Here are histograms of the $$p$$ values:

As you see, the $$p$$ values of the tests against a fully specified distribution are uniformly distributed, as they should be, but the ones from a fitted distribution are anything but.

n_sims <- 1e4
nn <- 20

pp_estimated <- pp_specified <- rep(NA,n_sims)
pb <- winProgressBar(max=n_sims)
for ( ii in 1:n_sims ) {
setWinProgressBar(pb,ii,paste(ii,"of",n_sims))
set.seed(ii)
sim <- rnorm(nn)
pp_specified[ii] <- ks.test(sim,y="pnorm",mean=0,sd=1)$$p pp_estimated[ii] <- ks.test(sim,y="pnorm",mean=mean(sim),sd=sd(sim))$$p
}
close(pb)

opar <- par(mfrow=c(1,2))
hist(pp_specified,main="Parameters specified",xlab="",col="lightgray")
hist(pp_estimated,main="Parameters estimated",xlab="",col="lightgray")
par(opar)

If your hypothesized reference distribution is normal, but you need to estimate the mean and variance, then the Lilliefors test would be appropriate. Other approaches may work for other distribution types. You may want to ask a specific question for the distribution type you are interested in.

I do not know of general framework for tests for fitted distributions. (As an extreme example, you could always use the empirical distribution of the data you observe. Of course, the fit would be perfect. But this would also likely not be very informative.)

EDIT - I just asked the general question here: Goodness of fit to a fitted distribution.