15
$\begingroup$

The software that I am currently using to build a model compares a "current run" model to a "reference model" and reports (where applicable) both a chi-squared p-value based on likelihood ratio tests and AIC values for each model. I know that one advantage of AIC over likelihood ratio tests is that AIC can be compared on non-nested models. However, I am not aware of any reason why AIC couldn't or shouldn't be compared on nested models. In my model, when comparing nested models for variable selection, I'm finding several cases where the likelihood ratio test and the AIC comparison are suggesting opposite conclusions.

Since both are based on likelihood calculations, I'm struggling to interpret these results. However the documentation of my software says (without explaining),

"If two models are nested (i.e. one is a sub-set of the other) then the more usual chi-squared test is the most appropriate to use. If the models are not nested, the AIC can be used..."

Can anyone elaborate on this and/or explain why AIC is not as helpful as likelihood ratio tests on nested models?

Improve this question
$\endgroup$
5
  • $\begingroup$ The best answer I can think of is that the likelihood ratio test is an actual test, and tests for the statistical significance of the added variable being tested in the nested model. So when appropriate, one might favor LRT more. $\endgroup$
    – Greenparker
    Feb 26, 2016 at 20:57
  • $\begingroup$ Seems to be answered her: stats.stackexchange.com/questions/20441/… $\endgroup$
    – Greenparker
    Feb 26, 2016 at 21:08
  • $\begingroup$ user42719, have I answered your question, or is there something else you need to know? $\endgroup$
    – Richard Hardy
    Mar 1, 2016 at 8:30
  • 3
    $\begingroup$ No reason for doing model selection vs. specifying a full model and using it was given. And note that AIC is just using a different $\alpha$-level cutoff for stepwise model building, so using AIC does not solve the serious problems with stepwise analysis. $\endgroup$
    – Frank Harrell
    May 8, 2016 at 18:16
  • $\begingroup$ @Greenparker It doesn't answer the question because the link is about nested and non-nested. Here, we care only nested. $\endgroup$
    – SmallChess
    Feb 16, 2017 at 2:36

1 Answer 1

Reset to default
30
$\begingroup$

AIC and likelihood ratio test (LRT) have different purposes.

  • AIC tells you whether it pays to have a richer model when your goal is to approximate the underlying data generating process the best you can in terms of Kullback-Leibler distance.
  • LRT tells you whether at a chosen confidence level you can reject the hypothesis that some restrictions on the richer model hold (e.g. some elements in the richer model are redundant).

You would use AIC if your goal is model selection for forecasting. You would use likelihood ratio test for significance testing. Different goals call for different tools.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Say the P-value of LRT for nested models is <0.05, which suggests that the additional variables added are not redundant, according to your explanation. However, the AIC of the richer model is larger than the AIC of the less rich model, which suggests that the preference goes against the richer model and the additional elements don't contribute to the richer model. this is contradictory. in this case, which model would you choose and how would you explain your selection? $\endgroup$
    – Zhoufeng
    Aug 12, 2021 at 0:09
  • $\begingroup$ @Zhoufeng, this is not contradictory because AIC and LRT answer different questions. It may come as a surprise the first time one encounters this. On the other hand, I do not think this is likely. Typically, the significance threshold that corresponds to AIC is about 15.7%, not 5%. So you may have improvement of AIC by including a variable that is not significant at 5% level (but significant at 15.7% level) but not vice versa. I say typically because I am not sure if there can be degenerate cases where the rule does not apply. $\endgroup$
    – Richard Hardy
    Aug 12, 2021 at 5:22
  • $\begingroup$ @Zhoufeng, regarding model choice, I would not use LRT for it. LRT is good for testing a hypothesis when the model is given but not for selecting a model among multiple candidates; see Hyndman "Why I don't like statistical tests". I would use AIC if I intended to use the model for forecasting/prediction. $\endgroup$
    – Richard Hardy
    Aug 12, 2021 at 5:25
  • $\begingroup$ Thanks for your reply. According to the definition of LRT, it is used for comparing nested models. So why cannot I use it for nested model selection? Also, It is kind of weird to me that the AIC is larger in the full model, compared to the sub-model, and the additional variable in the full model is the one with the lowest AIC, when all the variables are fitted individually. How could that happen? is it because that the AIC adds a penalty too hard once the number of variables increases? $\endgroup$
    – Zhoufeng
    Aug 12, 2021 at 6:46
  • $\begingroup$ @Zhoufeng, consider opening a new thread and asking a new question that concerns the specifics of your application. To answer briefly, AIC penalty is just fine for prediction but too soft (not too hard) for consistent model selection. Regarding the definition of LRT, I would be surprised if model selection was in it. Comparison, yes, as is implicit in any test. $\endgroup$
    – Richard Hardy
    Aug 12, 2021 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.