Let's say I have criteria A to select 1000 variables of interest and run a 1000 hypothesis tests on these.

However, I could use criteria B to select another 500 variables from a larger list of variables (e.g. 10000 which includes the original 1000) and add them to the original 1000 variables.

Then I can run 1500 hypothesis tests on the 1500 variables.

My question is the following: in the first situation I would correct for multiple testing using Bonferroni correction by using threshold alpha/1000. In the second scenario, however, theoretically I should use threshold alpha/1500. However, as the list of variables was obtained from a full 10000 variables list, would it be theoretically justified to correct by using threshold alpha/10000? Wouldn't the p-value be too stringent in this scenario and lead to more false negatives?

My understanding is that the event(selecting variables for hypothesis testing) and the event(hypothesis testing and correction for multiple testing) are independent thus threshold should be alpha/1500. However, in my attempt to avoid even one false positive, I want to find if there is theoretical justification for using alpha/10000? Any references? Any suggestions for more literature I can have a look at?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.