How should I adjust for the number of tests I do when using the likelihood ratio difference test in a model estimated using maximum likelihood (or some flavor of), where the null hypothesis of the test is that two models do not differ?

I am talking about family-wise adjustment for the number of tests.

I am a bit confused, because usually you intend to provide evidence for the alternative hypothesis. Here it is the other way around. Normally, if I would use (for example) the Bonferroni correction I could roughly divide the significance level by the number of tests.

If I do it this way, then the number of tests isn't really 'punishing' me, but rather making it easier to fail to reject the null hypothesis, which I intend to do. Let me clarify: let's say I set my significance at 5% (0.05) and I already conducted 1 test. For the next test I do, this becomes 0.05/2 = 0.025. So, to reject the null in the second test, is harder. However, the whole point of the chi-square difference test, is to 'fail to reject' the null. This makes it counter intuitive for me to adjust for multiple testing like this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.