I am interested in comparing two logistic regression models. The two models are nested: model 1 contains all predictors, and model 2 contains all predictors except 1. My goal is to test if removing that one variables makes the modelling significantly worse. I understand that I can use the likelihood ratio test (LRT) for the comparison.

But I'm wondering if I can use the Akaike Information Criterion (AIC) to compare the two models. Use of AIC seems more elegant, and gives stronger results in the study that I am looking at. On the other hand, I see that LRT is specifically designed for nested models.

Is it acceptable to use AIC to compare the two nested models, or is the LRT test more accurate than AIC when the models are nested?

Thank you, Fijoy

  • $\begingroup$ What do you mean by "stronger results"? $\endgroup$ – Glen_b Sep 9 '14 at 1:20
  • $\begingroup$ The p-value I get with LRT is 0.04, which is significant, but very close to the threshold of 0.05, so someone can doubt the result. The AIC of the full model is lower, and this is a cleaner result. Actually, AIC of all models that include that one variable in question is lower. $\endgroup$ – Fijoy Vadakkumpadan Sep 9 '14 at 1:29

When comparing models which differ by a single parameter, AIC corresponds to comparing by using an asymptotic (chi-squared) LRT on the additional parameter with $\alpha=15.73\%$.

Consider that with AIC you say the predictor should remain if AIC goes down when you include it, while with the LRT you say the predictor should remain if the chi-squared p-value is below $\alpha$.

Now $\text{AIC} = -2 \log\cal{L} + 2p$, so the AIC goes down if $-2 \log \cal{L}_0+2p_0>-2 \log \cal{L}_1+2p_1$ (where the subscript $0$ denotes the smaller model).

That is, it goes down if $-2 (\log \cal{L}_0- \log \cal{L}_1)>2(p_1-p_0)$.

Which is to say, with AIC you choose the larger model when $-2 (\log \cal{L}_0- \log \cal{L}_1)>2\times 1$. But the LHS is just the quantity used in the asymptotic chi-squared LRT, so under the null you choose the larger model when a $\chi^2_1>2$, which corresponds to using the asymptotic chi-squared test at a significance level of $0.1573$ (the tail area beyond $2$ in a $\chi^2_1$).

So your "clearer" result is nothing more than saying to the person who doubts* $p=0.04$ "well, if I use $\alpha=0.1573$, instead of $\alpha=0.05$, that $p=0.04$ looks pretty good". If they're capable of doing the calculation I just did for you, they may not be all that impressed by a desire to simply inflate a significance level because it makes the result look superficially (but not actually) "stronger".

* (perhaps somewhat inexplicably, as if p-values are all that meaningful)

[That's not to denigrate AIC in particular - indeed it may make more sense to actually use it than to use $p=0.05$ when deciding whether the additional variable is useful, but we should be clear about what it is we're choosing when we do it. If we choose it, our justification should not be based on this somewhat illusory notion of greater strength, but on its properties.]

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.