8
$\begingroup$

I'm trying to understand the Kolmogorov-Smirnov test using a very simple example. I generate a set of random, uniform values between 0 and 1.0. I then test that these values are from a uniform distribution by using the scipy kstest function. I'm expecing a very small D value and a pvalue close to 1.0, but instead I get wildly varying pvalues every time I run the code. What am I missing?

import numpy as np
import scipy
a = np.random.uniform(size=4999)
print(scipy.stats.kstest(a, 'uniform'))

Here are the outputs of a few consecutive runs:

(0.0075523161200627964, 0.93798952050647577)
(0.013787195268362473, 0.29799260741344774)
(0.014359046616557847, 0.25402403230845855)
(0.012521820948675988, 0.41329007558099806)
(0.011159003477582918, 0.56216895575676396)
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ You say widely varying p-values but are they all non-significant? $\endgroup$
    – Dan
    Commented Aug 27, 2014 at 16:59
  • 2
    $\begingroup$ You are getting what you should. Look at this CrossValidated thread $\endgroup$
    – Aniko
    Commented Aug 27, 2014 at 18:27

1 Answer 1

Reset to default
8
$\begingroup$

For the KS test the p-value is itself distributed uniformly in [0,1] if the H0 is true (which it is if you test whether it your sample is from $U(0,1)$ and the random number generation works okay). It therefore must "vary wildly" between 0 and 1, in fact its standard deviation is $1/\sqrt{12}$ which is roughly 0.3.

You can check this by looking whether the percentages of p values smaller or equal to some $p_0$ over your independent consecutive runs is close to said $p_0$.

See also Why are p-values uniformly distributed under the null hypothesis?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Is there any advantage of choosing small $p_0$, like 1e-3 vs 0.5? $\endgroup$
    – Ahmed Fasih
    Commented Apr 24, 2017 at 3:07
  • 1
    $\begingroup$ @ahmed No, in what I wrote, there is no significance to the value of $p_0$ at all. The percentages of p values smaller or equal to some $p_0$ should be close to $p_0$ regardless of the concrete $p_0$. That said, for very small $p_0$ one can encounter numeric problems when calculating the p-values. $\endgroup$
    – Momo
    Commented May 11, 2017 at 15:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.