# A non-uniform distribution of $p$-values…again

There appear to be many multiple-hypothesis-testing situations that result in a non-uniform distribution of $p$-values. I would like to describe one simple numerical experiment, and understand why it does not lead to the 'expected' result.

Suppose $X$ is a Rayleigh random variable (with shape parameter $b$). It can be shown that the random variable $Y=X/\text{mean}(X)$ has the following pdf:

$P(Y)=(\pi/2) Y \exp[(-\pi/4)Y^2]$,

which I'll call the SR (for standardized Rayleigh) distribution. Note that the SR distribution is independent of the parameter $b$. Note also that the SR distribution is equivalent to a Rayleigh distribution with shape parameter $b=\sqrt{2/\pi}$.

The numerical experiment is as follows: I generate 1000 Rayleigh random numbers using an arbitrary value of $b$ and then standardize these numbers by dividing each number by the overall sample mean. I then perform a KS goodness-of-fit test (where the null hypothesis is that the data is drawn from an SR distribution) and save the $p$-value. The preceding is repeated multiple times (e.g., 1000 times). A histogram of the resulting 1000 $p$-values is plotted.

Below is the MATLAB code. Question: Why are the $p$-values not uniformly distributed?

SR=makedist('Rayleigh','b',sqrt(2/pi));

for p_counter=1:1000
X=random('Rayleigh',11,1000,1);
Y=X/mean(X);
[h_ks(p_counter),p_ks(p_counter)]=kstest(Y,'Alpha',0.01,'CDF',SR);
end

histogram(p_ks);


When you write this:

Suppose X is a Rayleigh random variable (with shape parameter b). It can be shown that the random variable Y=X/mean(X)

You're talking about dividing by the population mean $\mu_X$, a fixed constant, giving $Y_i=X_i/\mu_X$.

Then in your experiment, you divide by the sample mean:

standardize these numbers by dividing each number by the overall sample mean.

This changes the distribution. The distribution of $X_i/\bar{X}$ is not Rayleigh.

More specifically, I think you should tend to see fewer large deviations from the Rayleigh cdf in this statistic than you see with the one that actually has the standard Rayleigh distribution, because the sample estimate will produce a fitted cdf that's "closer to the data" than the true one; as a result dividing by that estimate produces a standardized ecdf that's closer to the hypothetical distribution than you'd get if you divided by the population mean.

As a result I'd expect you should get an excess of large p-values and a deficit of small ones.

Your results are pretty much exactly what I'd have anticipated.

This effect is well known; we see it in other distributions. It's why the Lilliefors test* has smaller critical values than the Kolmogorov-Smirnov (which has no estimated parameters). The general idea is to use the same "largest difference in cdf" test statistic as with the Kolmogorov Smirnov but with the "theoretical" cdf being based on one or more estimated parameters (or equivalently, scaling the sample to some standard form using estimated parameters).

* (unfortunately, the text of the Wikipedia article at the link presently suggests the Lilliefors test is only for normality, but he also covered the exponential case, as you can see in the "Sources" section at the bottom of the article)

You could actually use the exponential-version of the Lilliefors test [1] for the Rayleigh distribution -- since the square of a Rayleigh random variate is exponential, you can just square your original data and test for exponential. (In this case you'd be dividing the squared data by its mean, not squaring your scaled values.)

Note that the asymptotic 5% critical value for the Kolmogorov-Smirnov is $1.36/\sqrt{n}$ while that for the Lilliefors when testing the exponential is $1.077/\sqrt{n}$ (i.e. as I suggested above, dividing an exponential sample by its mean produces a scaled ecdf which tends to be closer to the hypothetical than if you divided by the population mean).

[You could obtain critical values (and/or p-values) for a Lilliefors test using simulation under the null hypothesis. This is what Lilliefors actually did, but his simulation-sizes were pretty small (it was the 1960s, so computing facilities were limited) -- so you'd probably want to redo the simulation, particularly if you want p-values. If critical values are sufficient, there are more recent/more accurate tables available]

Added in edit: After a bit of googling around, it looks like the idea of using Lilliefors test (for the exponential) to test for Rayleigh (after transforming) was discussed in Edgeman & Scott (1987) [2].

[1] Lilliefors, H. (1969),
"On the Kolmogorov–Smirnov test for the exponential distribution with mean unknown",
Journal of the American Statistical Association, Vol. 64 . pp. 387–389.

[2] R.L. Edgeman, R.C. Scott (1987),
"Lilliefors's test for transformed variables",
Brazilian Journal of Probability and Statistics, 1, 101–112.

• Your point about the distribution of $X/\mu_X$ being different from $X/\bar{X}$ is well-taken. This does however lead me to a related question about the Rayleigh distribution which I will post separately. – user6006085 Mar 15 '16 at 19:06
• @user6006085: If you find this answer useful, please don't forget to upvote it by clicking on an arrow next to it. If it actually settles your question for you, you should additionally "accept" it by clicking on a green tick sign. Thanks. – amoeba says Reinstate Monica Mar 15 '16 at 21:49
• @user6006085 Knowing the distribution of $\bar{X}$ (while potentially interesting) won't help you with this problem though. I've made some edits here with more information. – Glen_b -Reinstate Monica Mar 15 '16 at 22:26
• Knowing the distribution of $\bar{X}$ would be useful as a first step towards obtaining the distribution of ${X}/\bar{X}$. If I knew the latter I would have the correct null distribution for the KS test, and presumably then everything would be kosher. – user6006085 Mar 16 '16 at 2:15
• It's unlikely to be tractable. The exponential case is simpler (I can tell you the distribution for $\bar{X}$ immediately, for example, and I even know the distribution of $X_i/\bar{X}$) ... and yet Lilliefors used simulation for that case and I am quite sure he knew those distributions as well. Why would he do that rather than just do it algebraically? An example of the kind of thing you might have missed considering is that those $X_i/\bar{X}$ values are not independent of each other. It's not so easy. – Glen_b -Reinstate Monica Mar 16 '16 at 2:58