The likelihood-ratio test is the optimal test for comparing the goodness-of-fit of two models. Is there some similar test that would allow me to test three models, or should I just compare models (A, B), (A, C), (B, C) with three ordinary likelihood-ratio tests?

  • $\begingroup$ The premise of the question is false: "The likelihood-ratio test is the optimal test for comparing the goodness-of-fit of two models" is not true, since a likelihood ratio test doesn't "compare two models", rather it compares a null model against an alternative model, and is treats them quite differently. The test statistic has the null distribution when the null is true. As a result you'd have to specify what you really mean when you say "compare three models". $\endgroup$
    – Glen_b
    Sep 21 '15 at 5:39

I haven't heard about such a test. The likelihood-ratio test requires nested models, so you either have a situation in which B and C are nested within A, or B is nested within A and C is nested within B. In the first case, you should compare (A,B) and (A,C). If neither B and C are better than A, pick A. If one of them is better than A, pick that one. If both are better than A, pick the one with lowest AIC or BIC.

In the second case, where C is nested in B, and both C and B are nested within A, start by comparing (B,C). If C is better, pick C. If there is no difference, test (A,C):

  1. If C is better, run (A, B). If there is no difference between A and B, pick C. Else, if B is better than A but C is not better than B (but better than A), pick B.

  2. If C is not better than either B or A, test (A,B). If B is better, pick B, and if neither B nor C is better than A, then just pick A.

If the models aren't nested, you shouldn't compare them using a likelihood-ratio test. I think there is another test based on relative likelihood that can be used, but I'm not familiar with it.

EDIT: Just a quick explanation of nested models.

Assume you have this regression model:

M1 <- lm(y ~ x1 + x2)

The model consists of a dependent variable and two independent variables (and of course intercept and error, but that's not important here). If you now consider this model:

M2 <- lm(y ~ x1)

This could be seen as a special case of the first model, in which the coefficient for x2 is 0. So no matter what value x2 takes, it doesn't affect y. M2 is thus nested within M1, because you can create M2 by setting one or more parameters in M1 to 0. Now consider this model:

M3 <- lm(y ~ x1 + x3)

M3 is not nested within M1 (or M2 of course) because it adds a new variable, x3. You cannot change a parameter in M1 to create M3 the same way as we did with M2. However, you can create M2 from M3 because the coefficient for x3 can be set to 0, and then we have M2! If we now create the following model:

M4 <- lm(y ~ x3)

This model is nested within M3, but not within M1 or M2, and neither M1 nor M2 is nested within M3.

This reasoning can also be extended to random effects in linear mixed models, and also to non-linear variables (a linear function is nested within a polynomial function or a spline function) etc.

  • $\begingroup$ I don't understand the thing about nested models. In Wikipedia, the likelihood-ratio test is described as follows: "The test is based on the likelihood ratio, which expresses how many times more likely the data are under one model than the other". I would like to compare three such models. $\endgroup$
    – emo
    Sep 20 '15 at 19:25
  • $\begingroup$ Oh the thing about nested models is also mentioned there. I guess I need to do some reading. $\endgroup$
    – emo
    Sep 20 '15 at 19:26
  • $\begingroup$ The AIC might also be used to select among non-nested models, although there is some dispute as noted on that page. You won't get things like p-values with AIC, however. Also see this Cross Validated page among many others on this site. $\endgroup$
    – EdM
    Sep 20 '15 at 19:33
  • $\begingroup$ @EdM: Good info about AIC and likelihood-ratio test in that link, thank you! Erno: I added a brief explanation of nested models. $\endgroup$
    – JonB
    Sep 20 '15 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.