27
$\begingroup$

I'm using AIC (Akaike's Information Criterion) to compare non-linear models in R. Is it valid to compare the AICs of different types of model? Specifically, I'm comparing a model fitted by glm versus a model with a random effect term fitted by glmer (lme4).

If not, is there a way such a comparison can be done? Or is the idea completely invalid?

|cite|improve this question
$\endgroup$
12
$\begingroup$

It depends. AIC is a function of the log likelihood. If both types of model compute the log likelihood the same way (i.e. include the same constant) then yes you can, if the models are nested.

I'm reasonably certain that glm() and lmer() don't use comparable log likelihoods.

The point about nested models is also up for discussion. Some say AIC is only valid for nested models as that is how the theory is presented/worked through. Others use it for all sorts of comparisons.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ My understanding is that lme4, by default uses REML where glm uses ML. They might be comparable if you made lmer use ML by setting REML = FALSE. $\endgroup$ – russellpierce Nov 29 '10 at 20:46
  • $\begingroup$ In addition to your Gavin's comment, it also depends what do you want to do with the model. Is the model for prediction or Thomas is looking for parsimony ? (I think) $\endgroup$ – suncoolsu Nov 29 '10 at 20:52
  • 3
    $\begingroup$ @drnexus: I don't think that is sufficient; you have to be sure that the same normalising constant is being applied in the log likelihood calculation. $\endgroup$ – Reinstate Monica - G. Simpson Nov 29 '10 at 20:52
  • 2
    $\begingroup$ @Thomas: for that you'd need to look at the code or speak to the person who wrote it to be sure. In general, assume the likelihoods aren't comparable across different software/packages/functions. $\endgroup$ – Reinstate Monica - G. Simpson Nov 29 '10 at 22:46
  • 1
    $\begingroup$ @user3490 Depends on the software and the algorithm used to get the estimates. In general I would presume they weren't the same unless I knew for certain that they were. $\endgroup$ – Reinstate Monica - G. Simpson Aug 20 '12 at 8:33
4
$\begingroup$

This is a great question that I've been curious about for a while.

For models in the same family (ie. auto-regressive models of order k or polynomials) AIC/BIC makes a lot of sense. In other cases it's less clear. Computing the log-likelihood exactly (with the constant terms) should work, but using more complicated model comparison such as Bayes Factors is probably better (http://www.jstor.org/stable/2291091).

If the models have the same loss/error-function one alternative is to just compare the cross-validated log-likelihoods. That's usually what I try to do when I'm not sure AIC/BIC makes sense in a certain situation.

|cite|improve this answer
$\endgroup$
0
$\begingroup$

Note that in some cases AIC cannot even compare models of the same type, like ARIMA models with a different order of differencing. Quoting Forecasting: Principles and Practice by Rob J Hyndman and George Athanasopoulos:

It is important to note that these information criteria tend not to be good guides to selecting the appropriate order of differencing ($d$) of a model, but only for selecting the values of $p$ and $q$. This is because the differencing changes the data on which the likelihood is computed, making the AIC values between models with different orders of differencing not comparable. So we need to use some other approach to choose $d$, and then we can use the AICc to select $p$ and $q$.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Indeed, but a crucial point is that it is not the type of model that makes the comparison problematic, it is the data on which the likelihood is defined. $\endgroup$ – Richard Hardy Jun 24 '19 at 8:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.