Suppose I want to test for a significant effect of an explanatory variable on five different binary outcomes (observed in the same subjects from the same population). The outcomes are not independent of one another.

For each of the outcomes I can fit a logistic regression model and obtain a p-value for the test that the explanatory variable has non-zero beta.

Suppose the observed p-values for these five tests, in increasing order, are:

p1 = 0.0113
p2 = 0.0402
p3 = 0.0811
p4 = 0.1134
p5 = 0.3397

I cannot just consider two of the effects to be statistically significant at the 0.05 level - I need to allow somehow for multiple testing, i.e. five tests having been performed.

Using a Bonferroni correction I would compare all tests against 0.05/5 = 0.01 and conclude none are significant. Using the Benjamini-Hochberg procedure to control the FDR at 0.05 I would compare the p-values to 0.01, 0.02, 0.03, 0.04, 0.05 respectively, and again conclude that none are significant after multiple testing correction.

But due to the relationships between the outcomes I feel these methods are too conservative - we don't have five independent tests. So I want to use a permutation test to preserve the correlation structure across tests.

The way I would do this is:

  • Shuffle explanatory variable among subjects
  • Perform logistic regression test on each outcome, generating five p-values
  • Repeat this 100,000 times

I would then consider that the set

{ smallest p-value in permutation i, i = 1-100,000 }

forms a null distribution for my smallest observed p-value, the set

{ second smallest p-value in permutation i, i = 1-100,000 }

forms a null distribution for my second smallest observed p-value, and so on.

So I would compare my five observed p-values against the empirical 0.05 level of each of the five null distributions I have generated. After performing the permutations, these empirical 0.05 levels evaluate to:

thresh1 = 0.01272858
thresh2 = 0.05610931
thresh3 = 0.13864864
thresh4 = 0.26762228
thresh5 = 0.48448522

Clearly each of my observed p-values is below the corresponding empirical 0.05 threshold. In fact, the adjusted "empirical" p-values (proportion of each null that is less than or equal to my observed p-values) evaluate to:

p*1 = 0.04434
p*2 = 0.03090
p*3 = 0.01922
p*4 = 0.00772
p*5 = 0.01595

These are all below 0.05. But surely this doesn't mean that I can declare a significant effect for all five outcomes?

In particular my biggest original p-value was 0.3397. I appreciate that this is significantly low in being the largest of five observed p-values. But it does not reach nominal significance in itself: it is not even close to the unadjusted 0.05 p-value threshold! So for which outcomes can I declare a significant effect?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.