7
$\begingroup$

This question already has an answer here:

I'm trying to understand the Kolmogorov-Smirnov test using a very simple example. I generate a set of random, uniform values between 0 and 1.0. I then test that these values are from a uniform distribution by using the scipy kstest function. I'm expecing a very small D value and a pvalue close to 1.0, but instead I get wildly varying pvalues every time I run the code. What am I missing?

import numpy as np
import scipy
a = np.random.uniform(size=4999)
print(scipy.stats.kstest(a, 'uniform'))

Here are the outputs of a few consecutive runs:

(0.0075523161200627964, 0.93798952050647577)
(0.013787195268362473, 0.29799260741344774)
(0.014359046616557847, 0.25402403230845855)
(0.012521820948675988, 0.41329007558099806)
(0.011159003477582918, 0.56216895575676396)
$\endgroup$

marked as duplicate by whuber Aug 27 '14 at 20:14

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ You say widely varying p-values but are they all non-significant? $\endgroup$ – Dan Aug 27 '14 at 16:59
  • 2
    $\begingroup$ You are getting what you should. Look at this CrossValidated thread $\endgroup$ – Aniko Aug 27 '14 at 18:27
7
$\begingroup$

For the KS test the p-value is itself distributed uniformly in [0,1] if the H0 is true (which it is if you test whether it your sample is from $U(0,1)$ and the random number generation works okay). It therefore must "vary wildly" between 0 and 1, in fact its standard deviation is $1/\sqrt{12}$ which is roughly 0.3.

You can check this by looking whether the percentages of p values smaller or equal to some $p_0$ over your independent consecutive runs is close to said $p_0$.

See also Why are p-values uniformly distributed under the null hypothesis?

$\endgroup$
  • $\begingroup$ Is there any advantage of choosing small $p_0$, like 1e-3 vs 0.5? $\endgroup$ – Ahmed Fasih Apr 24 '17 at 3:07
  • 1
    $\begingroup$ @ahmed No, in what I wrote, there is no significance to the value of $p_0$ at all. The percentages of p values smaller or equal to some $p_0$ should be close to $p_0$ regardless of the concrete $p_0$. That said, for very small $p_0$ one can encounter numeric problems when calculating the p-values. $\endgroup$ – Momo May 11 '17 at 15:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.