I have a question that's very similar to this question: Combine multiple independent permutation tests

My 'issue' is exactly the same: I have 20 subjects and I used permutation tests to determine the correlation between one measure and another, giving me 20 test-statistics/pvalues that I want to combine in one single p-value. Note that the actual case is a bit more complicated than this, so mixed effects modelling or other methods to analyse the data in one go are not appropriate in this case.

The null hypothesis (I think) I want to test is that the 20 test statistics / pvalues combined are not different from a random uniform distribution. The alternative hypothesis is that the test statistics combined are larger than what can be expected by chance (or that the pvalues are lower than can be expected by chance). What can I do to test this hypothesis?

The permutations I have been doing are not computationally very demanding, so that's not a limiting factor in terms of possible ways to address this.

NOTE: In the answer to the question cited above, it was suggested to use Fisher's method as a way of combining p values, but I'm not sure whether that's the right test to use in this case, since it tests the null hypothesis that all of separate null hypotheses are true. The alternative is that one of the alternative hypotheses are true, and that's to anti-conservative I think.

Thank you for your help!


If you believe that the signal is spread out over the tests as opposed to concentrated in a small number of them then Stouffer's method may be more suitable than Fisher's method.

Loughin's extensive simulations "A systematic comparison of methods for combining p--values from independent tests" available here may be of interest.

See also some of the answers to Test for significant excess of significant p-values across multiple comparisons


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.