Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm using AIC (Akaike's Information Criterion) to compare non-linear models in R. Is it valid to compare the AICs of different types of model? Specifically, I'm comparing a model fitted by glm versus a model with a random effect term fitted by glmer (lme4).

If not, is there a way such a comparison can be done? Or is the idea completely invalid?

share|improve this question

2 Answers 2

up vote 5 down vote accepted

It depends. AIC is a function of the log likelihood. If both types of model compute the log likelihood the same way (i.e. include the same constant) then yes you can, if the models are nested.

I'm reasonably certain that glm() and lmer() don't use comparable log likelihoods.

The point about nested models is also up for discussion. Some say AIC is only valid for nested models as that is how the theory is presented/worked through. Others use it for all sorts of comparisons.

share|improve this answer
    
My understanding is that lme4, by default uses REML where glm uses ML. They might be comparable if you made lmer use ML by setting REML = FALSE. –  rpierce Nov 29 '10 at 20:46
    
In addition to your Gavin's comment, it also depends what do you want to do with the model. Is the model for prediction or Thomas is looking for parsimony ? (I think) –  suncoolsu Nov 29 '10 at 20:52
3  
@drnexus: I don't think that is sufficient; you have to be sure that the same normalising constant is being applied in the log likelihood calculation. –  Gavin Simpson Nov 29 '10 at 20:52
2  
@Thomas: for that you'd need to look at the code or speak to the person who wrote it to be sure. In general, assume the likelihoods aren't comparable across different software/packages/functions. –  Gavin Simpson Nov 29 '10 at 22:46
1  
@user3490 Depends on the software and the algorithm used to get the estimates. In general I would presume they weren't the same unless I knew for certain that they were. –  Gavin Simpson Aug 20 '12 at 8:33

This is a great question that I've been curious about for a while.

For models in the same family (ie. auto-regressive models of order k or polynomials) AIC/BIC makes a lot of sense. In other cases it's less clear. Computing the log-likelihood exactly (with the constant terms) should work, but using more complicated model comparison such as Bayes Factors is probably better (http://www.jstor.org/stable/2291091).

If the models have the same loss/error-function one alternative is to just compare the cross-validated log-likelihoods. That's usually what I try to do when I'm not sure AIC/BIC makes sense in a certain situation.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.