A non-uniform distribution of $p$-values...again There appear to be many multiple-hypothesis-testing situations that result in a non-uniform distribution of $p$-values. I would like to describe one simple numerical experiment, and understand why it does not lead to the 'expected' result.
Suppose $X$ is a Rayleigh random variable (with shape parameter $b$). It can be shown that the random variable $Y=X/\text{mean}(X)$ has the following pdf:
$P(Y)=(\pi/2) Y \exp[(-\pi/4)Y^2]$,
which I'll call the SR (for standardized Rayleigh) distribution. Note that the SR distribution is independent of the parameter $b$. Note also that the SR distribution is equivalent to a Rayleigh distribution with shape parameter $b=\sqrt{2/\pi}$.
The numerical experiment is as follows: I generate 1000 Rayleigh random numbers using an arbitrary value of $b$ and then standardize these numbers by dividing each number by the overall sample mean. I then perform a KS goodness-of-fit test (where the null hypothesis is that the data is drawn from an SR distribution) and save the $p$-value. The preceding is repeated multiple times (e.g., 1000 times). A histogram of the resulting 1000 $p$-values is plotted.
Below is the MATLAB code. Question: Why are the $p$-values not uniformly distributed?
SR=makedist('Rayleigh','b',sqrt(2/pi));

for p_counter=1:1000
    X=random('Rayleigh',11,1000,1);   
    Y=X/mean(X);
    [h_ks(p_counter),p_ks(p_counter)]=kstest(Y,'Alpha',0.01,'CDF',SR);
end

histogram(p_ks);


 A: When you write this:

Suppose X is a Rayleigh random variable (with shape parameter b). It can be shown that the random variable Y=X/mean(X) 

You're talking about dividing by the population mean $\mu_X$,  a fixed constant, giving $Y_i=X_i/\mu_X$.
Then in your experiment, you divide by the sample mean:

standardize these numbers by dividing each number by the overall sample mean.

This changes the distribution. The distribution of $X_i/\bar{X}$ is not Rayleigh.
More specifically, I think you should tend to see fewer large deviations from the Rayleigh cdf in this statistic than you see with the one that actually has the standard Rayleigh distribution, because the sample estimate will produce a fitted cdf that's "closer to the data" than the true one; as a result dividing by that estimate produces a standardized ecdf that's closer to the hypothetical distribution than you'd get if you divided by the population mean. 
As a result I'd expect you should get an excess of large p-values and a deficit of small ones.
Your results are pretty much exactly what I'd have anticipated.
This effect is well known; we see it in other distributions. It's why the Lilliefors test* has smaller critical values than the Kolmogorov-Smirnov (which has no estimated parameters).  The general idea is to use the same "largest difference in cdf" test statistic as with the Kolmogorov Smirnov but with the "theoretical" cdf being based on one or more estimated parameters (or equivalently, scaling the sample to some standard form using estimated parameters).
* (unfortunately, the text of the Wikipedia article at the link presently suggests the Lilliefors test is only for normality, but he also covered the exponential case, as you can see in the "Sources" section at the bottom of the article)
You could actually use the exponential-version of the Lilliefors test [1] for the Rayleigh distribution -- since the square of a Rayleigh random variate is exponential, you can just square your original data and test for exponential. (In this case you'd be dividing the squared data by its mean, not squaring your scaled values.)
Note that the asymptotic 5% critical value for the Kolmogorov-Smirnov is $1.36/\sqrt{n}$ while that for the Lilliefors when testing the exponential is $1.077/\sqrt{n}$ (i.e. as I suggested above, dividing an exponential sample by its mean produces a scaled ecdf which tends to be closer to the hypothetical than if you divided by the population mean).
[You could obtain critical values (and/or p-values) for a Lilliefors test using simulation under the null hypothesis. This is what Lilliefors actually did, but his simulation-sizes were pretty small (it was the 1960s, so computing facilities were limited) -- so you'd probably want to redo the simulation, particularly if you want p-values. If critical values are sufficient, there are more recent/more accurate tables available]

Added in edit: After a bit of googling around, it looks like the idea of using Lilliefors test (for the exponential) to test for Rayleigh (after transforming) was discussed in Edgeman & Scott (1987) [2].
[1] Lilliefors, H. (1969),
"On the Kolmogorov–Smirnov test for the exponential distribution with mean unknown",
Journal of the American Statistical Association, Vol. 64 . pp. 387–389.
[2] R.L. Edgeman, R.C. Scott (1987),
"Lilliefors's test for transformed variables",
Brazilian Journal of Probability and Statistics, 1, 101–112.
