Non-uniform distribution of p-values I'm running a Monte Carlo simulation to check whether a distribution generated by my model fits a pre-defined target distribution where the parameters of the target distribution are estimated from the data.
I ran a KS-test on the output of my simulation against the target distribution and checked p-values, and they were consistently high (p~1), suggesting that the fit was indeed good.
Just as a sense check I generated ~10^3 samples to check the distribution of these p-values. From my understanding under the null hypothesis of identical disributions these p-values should be uniformly distributed.
However, oddly, the distribution of p-values is skewed towards 1 (see distribution of p-values below) - it seems an overly large number of the samples fit the target distribution closely.
I'm struggling to understand how to interpret this. Obviously if it was skewed towards 0 it's pretty clear that the samples don't fit the target. How should one interpret the inverse situation as described?

Edit: Added that parameters are estimated during the fitting process. This appears to explain the skew.
 A: As whuber has commented: the Kolmogorov-Smirnov test is only valid as a comparison against a fully specified distribution. You cannot use it to compare an observed distribution against a distribution whose parameters have been estimated based on your observed sample. If you do so, your p-values will not be uniformly distributed under the null hypothesis, but show the exact pattern you have observed.
This is unfortunately an extremely common error, which you can very often find in online tutorials.
As a little illustration, let us simulate $x_1, \dots, x_{20}\sim N(0,1)$, then run a K-S test first against a fully specified $N(0,1)$ distribution, then against an estimated $N(\hat{\mu},\hat{\sigma}^2)$ distribution, where $\hat{\mu}$ and $\hat{\sigma}^2$ are estimated based on $x_1, \dots, x_{20}$. Record the $p$ value. Do this 10,000 times. Here are histograms of the $p$ values:

As you see, the $p$ values of the tests against a fully specified distribution are uniformly distributed, as they should be, but the ones from a fitted distribution are anything but.
n_sims <- 1e4
nn <- 20

pp_estimated <- pp_specified <- rep(NA,n_sims)
pb <- winProgressBar(max=n_sims)
    for ( ii in 1:n_sims ) {
        setWinProgressBar(pb,ii,paste(ii,"of",n_sims))
        set.seed(ii)
        sim <- rnorm(nn)
        pp_specified[ii] <- ks.test(sim,y="pnorm",mean=0,sd=1)$p
  pp_estimated[ii] <- ks.test(sim,y="pnorm",mean=mean(sim),sd=sd(sim))$p
    }
close(pb)

opar <- par(mfrow=c(1,2))
    hist(pp_specified,main="Parameters specified",xlab="",col="lightgray")
    hist(pp_estimated,main="Parameters estimated",xlab="",col="lightgray")
par(opar)

If your hypothesized reference distribution is normal, but you need to estimate the mean and variance, then the Lilliefors test would be appropriate. Other approaches may work for other distribution types. You may want to ask a specific question for the distribution type you are interested in.
I do not know of  general framework for goodness-of-fit tests for fitted distributions. (As an extreme example, you could always use the empirical distribution of the data you observe. Of course, the fit would be perfect. But this would also likely not be very informative.)
EDIT - I just asked the general question here: Goodness of fit to a fitted distribution.
