Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm using AIC (Akaike's Information Criterion) to compare non-linear models in R. Is it valid to compare the AICs of different types of model? Specifically, I'm comparing a model fitted by glm versus a model with a random effect term fitted by glmer (lme4).
If not, is there a way such a comparison can be done? Or is the idea completely invalid?
It depends. AIC is a function of the log likelihood. If both types of model compute the log likelihood the same way (i.e. include the same constant) then yes you can, if the models are nested.
I'm reasonably certain that glm() and lmer() don't use comparable log likelihoods.
The point about nested models is also up for discussion. Some say AIC is only valid for nested models as that is how the theory is presented/worked through. Others use it for all sorts of comparisons.
This is a great question that I've been curious about for a while.
For models in the same family (ie. auto-regressive models of order k or polynomials) AIC/BIC makes a lot of sense. In other cases it's less clear. Computing the log-likelihood exactly (with the constant terms) should work, but using more complicated model comparison such as Bayes Factors is probably better (http://www.jstor.org/stable/2291091).
If the models have the same loss/error-function one alternative is to just compare the cross-validated log-likelihoods. That's usually what I try to do when I'm not sure AIC/BIC makes sense in a certain situation.
Note that in some cases AIC cannot even compare models of the same type, like ARIMA models with a different order of differencing. Quoting Forecasting: Principles and Practice by Rob J Hyndman and George Athanasopoulos:
It is important to note that these information criteria tend not to be good guides to selecting the appropriate order of differencing ($d$) of a model, but only for selecting the values of $p$ and $q$. This is because the differencing changes the data on which the likelihood is computed, making the AIC values between models with different orders of differencing not comparable. So we need to use some other approach to choose $d$, and then we can use the AICc to select $p$ and $q$.
