1
$\begingroup$

How should I adjust for the number of tests I do when using the likelihood ratio difference test in a model estimated using maximum likelihood (or some flavor of), where the null hypothesis of the test is that two models do not differ?

I am talking about family-wise adjustment for the number of tests.

I am a bit confused, because usually you intend to provide evidence for the alternative hypothesis. Here it is the other way around. Normally, if I would use (for example) the Bonferroni correction I could roughly divide the significance level by the number of tests.

If I do it this way, then the number of tests isn't really 'punishing' me, but rather making it easier to fail to reject the null hypothesis, which I intend to do. Let me clarify: let's say I set my significance at 5% (0.05) and I already conducted 1 test. For the next test I do, this becomes 0.05/2 = 0.025. So, to reject the null in the second test, is harder. However, the whole point of the chi-square difference test, is to 'fail to reject' the null. This makes it counter intuitive for me to adjust for multiple testing like this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.