The software that I am currently using to build a model compares a "current run" model to a "reference model" and reports (where applicable) both a chi-squared p-value based on likelihood ratio tests and AIC values for each model. I know that one advantage of AIC over likelihood ratio tests is that AIC can be compared on non-nested models. However, I am not aware of any reason why AIC couldn't or shouldn't be compared on nested models. In my model, when comparing nested models for variable selection, I'm finding several cases where the likelihood ratio test and the AIC comparison are suggesting opposite conclusions.

Since both are based on likelihood calculations, I'm struggling to interpret these results. However the documentation of my software says (without explaining),

"If two models are nested (i.e. one is a sub-set of the other) then the more usual chi-squared test is the most appropriate to use. If the models are not nested, the AIC can be used..."

Can anyone elaborate on this and/or explain why AIC is not as helpful as likelihood ratio tests on nested models?

  • $\begingroup$ The best answer I can think of is that the likelihood ratio test is an actual test, and tests for the statistical significance of the added variable being tested in the nested model. So when appropriate, one might favor LRT more. $\endgroup$ – Greenparker Feb 26 '16 at 20:57
  • $\begingroup$ Seems to be answered her: stats.stackexchange.com/questions/20441/… $\endgroup$ – Greenparker Feb 26 '16 at 21:08
  • $\begingroup$ user42719, have I answered your question, or is there something else you need to know? $\endgroup$ – Richard Hardy Mar 1 '16 at 8:30
  • 1
    $\begingroup$ No reason for doing model selection vs. specifying a full model and using it was given. And note that AIC is just using a different $\alpha$-level cutoff for stepwise model building, so using AIC does not solve the serious problems with stepwise analysis. $\endgroup$ – Frank Harrell May 8 '16 at 18:16
  • $\begingroup$ @Greenparker It doesn't answer the question because the link is about nested and non-nested. Here, we care only nested. $\endgroup$ – SmallChess Feb 16 '17 at 2:36

AIC and likelihood ratio test (LRT) have different purposes.

  • AIC tells you whether it pays to have a richer model when your goal is approximating the underlying data generating process the best you can in terms of Kullback-Leibler distance.
  • LRT tells you whether at a chosen confidence level you can reject the hypothesis that some restrictions on the richer model hold (e.g. some elements in the richer model are redundant).

You would use AIC if your goal is model selection for forecasting. You would use likelihood ratio test for significance testing. Different goals call for different tools.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.