Say I have multiple tests on N data points, for each I get a p-value ($0 < p < 1$). If the null hypothesis is true, I would expect that the distribution of p-value to be uniform. If, however the empiric CDF is larger than the uniform diagonal ($\text{CDF}(p) > p$), e.g. there are 0.1 fraction of samples with p-value below 0.01 etc., I could use a K-S test to test for the p-value distribution to be far from uniform, and thus conclude with one p-value to the question: "Are there significant point of data which deviates from the null hypothesis"?

Is the above methodology valid? Can you please give me a reference? If not? do you think this can be written as a method paper?

  • $\begingroup$ Interesting idea. I don't know the full answer. But unless you have a LOT of tests, it's going to be hard to compare two curves. And you would have to have a good interpretation of the KS test, which, as far as I know, tests the equality of two curves, not their equivalence at any particular points. $\endgroup$ – Peter Flom Oct 2 '13 at 10:33
  • 1
    $\begingroup$ I've seen the K-S test used in meta-analysis to assess whether p-values are uniformly distributed. More common is Fisher's method, presumably because it's more powerful against a directional alternative. $\endgroup$ – Scortchi - Reinstate Monica Oct 2 '13 at 10:59
  • $\begingroup$ I don't understand. I may not have though this trough but suppose the KS soundly rejects the null that the p-val are uniform because (e.g. in the sense that the maximum of the KS statistic is attained there) there are way too many tests which results in pvals>0.9 (and a corresponding through in pvals between 0.8 and 0.9). How are you interpreting this? $\endgroup$ – user603 Oct 2 '13 at 12:14
  • 1
    $\begingroup$ Also, for the KS test, I assume it's important that the pvals are iid. Is this the case? $\endgroup$ – user603 Oct 2 '13 at 12:22

The basic idea sounds OK to me, but you would need to specify what kind of uncertainty you try to quantify with your test. In the usual case of statistical testing the uncertainty is supposed to be due to random sampling from a population. In your case you could be trying to quantify the randomness that belongs to a Monte Carlo experiment. This is a good idea, but it is non-standard and thus needs to be discussed carefully. Also note that the distribution of $p$-values can legitimately deviate from a standard continuous uniform distribution even if the null hypothesis is true, e.g. in a one-sided test: Is there a sample distribution so that the distribution of p-value is skewed towards 1?

To give a very meta answer: the K-S test does not perform too well, as you can see by looking at the distribution of $p$-values as you can see in the following simulation using Stata:

set seed 12345
clear all
set more off

program define sim
    drop _all
    set obs 100
    gen x = runiform()
    ksmirnov x = x
simulate p=r(p) p_cor=r(p_cor), reps(20000) : sim

A K-S test on the resulting $p$-values results in the conclusion that the null hypothesis that the distribution of $p$-values is uniformly distributed is rejected at the 5% level:

. ksmirnov p = p

One-sample Kolmogorov-Smirnov test against theoretical distribution

 Smaller group       D       P-value  Corrected
 p:                  0.0001    0.999
 Cumulative:        -0.0221    0.000
 Combined K-S:       0.0221    0.000      0.000

Note: ties exist in dataset;
      there are 19982 unique values out of 20000 observations.

. ksmirnov p_cor = p_cor

One-sample Kolmogorov-Smirnov test against theoretical distribution

 Smaller group       D       P-value  Corrected
 p_cor:              0.0315    0.000
 Cumulative:        -0.0010    0.961
 Combined K-S:       0.0315    0.000      0.000

Note: ties exist in dataset;
      there are 19986 unique values out of 20000 observations.

For displaying the results graphically I like this graph: it shows on the y-axis the difference between the empirical estimate of the Cumulative Distribution Function (CDF) and the theoretical (continuous standard uniform) distribution. On the x-axis is the nominal p-value. The logic behind this graph is that for $p$-values in a simulation study in which the null hypothesis is true, the empirical CDF is an empirical estimate of the $p$-value. The empirical CDF gives for each nominal $p$-value an estimate of the probability of drawing a sample which deviates at least as much from the null hypothesis as the current sample (i.e. has a nominal $p$-value less than or equal to the current nominal $p$-value) if the null hypothesis is true. So negative values on the y-axis means that the emprical estimates of the $p$-value are less than the nominal $p$-values and positive values on the y-axis say that the empirical estimates of the $p$-values are larger than the nominal $p$-values.

label var p `""standard" "p-value""'
label var p_cor `""corrected" "p-value""'

simpplot p p_cor, overall reps(20000) ///
    scheme(s2color) ylab(,angle(horizontal))   

enter image description here

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.