2
$\begingroup$

I have a set of independent null hypothesis significance tests and their resulting p-values and I would like to make the claim that they are not all null by looking at the distribution of independent p-values. Since under the (meta) null hypothesis that all my hypotheses are actually null, the distribution of p-values is uniform, I though that I could simply perform a Kolmogorov-Smirnov test on that distribution and report the resulting p-value. Is this legitimate and is this something that any of you has seen done before? If so, would you have a reference?

I know of false discovery rate by Storey and Tibshirani but I would really wish to have a single p-value for the whole distribution rather than a set of q-values.

$\endgroup$
2
  • 3
    $\begingroup$ I would advise approaching this cautiously. First, many tests do not produce exact p-values: they are approximate. Second, others do produce excellent p-values only when the p-values are small. Third, a test statistic with a discrete distribution obviously cannot yield a uniform distribution of p-values. Fourth, you're concerned about a specific alternative hypothesis, where the density of the p-value distribution grows at the lower values and shrinks at the higher, so this calls for a specialized test. The first three imply that a test of uniformity might give misleading results. $\endgroup$
    – whuber
    Jun 29 '18 at 15:23
  • 1
    $\begingroup$ Have you seen stats.stackexchange.com/questions/20616? $\endgroup$
    – whuber
    Jun 29 '18 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.