I'm not an expert on the field, but I need to know more about how to combined multiple significance test p-values after a two sample Kolmogorov Smirnov (KS)test on two distributions.

But I have two cases: 1) a model where I can generate as many realizations I wish and 2) only one (or a few) observed realization.

1) I run multiple KS tests, each test gives me a p-value: I used the Fisher Method and everything goes as expected in literature Fisher tesi in CrossValidate and Fisher test in CrossValidated(unifiorm distribution of p-values if hypothesis cannot be reject).

Now, since I could complicate the model in different ways, I was wondering if I can also use the Bonferroni-Holm correction Wiki-page on method for the same dataset with multiple p-values (meta-analysis).

And which is actually the difference between Bonferrroni-type corrections and Fisher-type ones (or Stouffer type), more in terms when to use them, which are the requirements and conditions and limitations. For example multiple comparison is indented for different tests with different hypothesis, or inter-dependence among tests

2) what changes about the methods, if I have only 1 observed sequence and I use Bootstrapping o permutation test to artificially created several sequences and making again K-S stests and several p-values again.



As you point out there is clearly some connection here but there are also important differences.

The problem of combining $p$-values is poorly specified. This may account for the number of methods available and their differing behaviour. The null hypothesis $H_0$ is well defined, that all $p_i$ have a uniform distribution on the unit interval. There are two classes of alternative hypothesis

  • $H_A$: all $p_i$ have the same (unknown) non--uniform, non--increasing density,
  • $H_B$: at least one $p_i$ has an (unknown) non--uniform, non--increasing density.

If all the tests being combined come from what are basically replicates then $H_A$ is appropriate whereas if they are of different kinds of test or different conditions then $H_B$ is appropriate. There exists the possibility that the tests being combined may be very different for instance some tests of means, some of variances, and so on.

One feature of most, but not all, of the methods for combining $p$-values is that even if no individual $p$-value reaches a conventional significance level the overall $p$ may. There is extensive discussion of this in this Q&A Can a meta-analysis of studies which are all "not statistically signficant" lead to a "significant" conclusion?

An extensive simulation study comparing the methods for combining $p$-values by Loughin entitled "A systematic comparison of methods for combining p-values from independent tests" available here discusses which method is sensitive to which distribution and strength of evidence.

For the multiple comparison methods the issue is approached from a different persepective. In the usual situation all the tests have something in common and are considered as a family of tests. In such a case the adjustment methods try to ensure that the chances of a Type I error are maintained below the claimed size of the test. In such a case the methods will not claim significance unless some individual tests do.


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.