Cumulative distribution of p-value Say I have multiple tests on N data points, for each I get a p-value ($0 < p < 1$).
If the null hypothesis is true, I would expect that the distribution of p-value to be uniform. If, however the empiric CDF is larger than the uniform diagonal ($\text{CDF}(p) > p$), e.g. there are 0.1 fraction of samples with p-value below 0.01 etc., I could use a K-S test to test for the p-value distribution to be far from uniform, and thus conclude with one p-value to the question: "Are there significant point of data which deviates from the null hypothesis"?
Is the above methodology valid? Can you please give me a reference? If not? do you think this can be written as a method paper?
 A: The basic idea sounds OK to me, but you would need to specify what kind of uncertainty you try to quantify with your test. In the usual case of statistical testing the uncertainty is supposed to be due to random sampling from a population. In your case you could be trying to quantify the randomness that belongs to a Monte Carlo experiment. This is a good idea, but it is non-standard and thus needs to be discussed carefully. Also note that the distribution of $p$-values can legitimately deviate from a standard continuous uniform distribution even if the null hypothesis is true, e.g. in a one-sided test: Is there a sample distribution so that the distribution of p-value is skewed towards 1?
To give a very meta answer: the K-S test does not perform too well, as you can see by looking at the distribution of $p$-values as you can see in the following simulation using Stata:
set seed 12345
clear all
set more off

program define sim
    drop _all
    set obs 100
    gen x = runiform()
    ksmirnov x = x
end
simulate p=r(p) p_cor=r(p_cor), reps(20000) : sim

A K-S test on the resulting $p$-values results in the conclusion that the null hypothesis that the distribution of $p$-values is uniformly distributed is rejected at the 5% level:
. ksmirnov p = p

One-sample Kolmogorov-Smirnov test against theoretical distribution
           p

 Smaller group       D       P-value  Corrected
 ----------------------------------------------
 p:                  0.0001    0.999
 Cumulative:        -0.0221    0.000
 Combined K-S:       0.0221    0.000      0.000

Note: ties exist in dataset;
      there are 19982 unique values out of 20000 observations.

. ksmirnov p_cor = p_cor

One-sample Kolmogorov-Smirnov test against theoretical distribution
           p_cor

 Smaller group       D       P-value  Corrected
 ----------------------------------------------
 p_cor:              0.0315    0.000
 Cumulative:        -0.0010    0.961
 Combined K-S:       0.0315    0.000      0.000

Note: ties exist in dataset;
      there are 19986 unique values out of 20000 observations.

For displaying the results graphically I like this graph: it shows on the y-axis the difference between the empirical estimate of the Cumulative Distribution Function (CDF) and the theoretical (continuous standard uniform) distribution. On the x-axis is the nominal p-value. The logic behind this graph is that for $p$-values in a simulation study in which the null hypothesis is true, the empirical CDF is an empirical estimate of the $p$-value. The empirical CDF gives for each nominal $p$-value an estimate of the probability of drawing a sample which deviates at least as much from the null hypothesis as the current sample (i.e. has a nominal $p$-value less than or equal to the current nominal $p$-value) if the null hypothesis is true. So negative values on the y-axis means that the emprical estimates of the $p$-value are less than the nominal $p$-values and positive values on the y-axis say that the empirical estimates of the $p$-values are larger than the nominal $p$-values.
label var p `""standard" "p-value""'
label var p_cor `""corrected" "p-value""'

simpplot p p_cor, overall reps(20000) ///
    scheme(s2color) ylab(,angle(horizontal))   


