I am comparing three relatively simple GLMs having a Gamma distribution with AIC and BIC. The aim is to identify the effects of fertilizers (fdung), year and site on biomass of a specific grass species. Hence, the aim is not to predict new values but merely to identify the effects of the three factors.

Here are the models used:

res1 <-  glm((Biomass..g.m².) ~  fdung * fyear * fblock, family=Gamma(link="identity"))
res2 <-  glm((Biomass..g.m².) ~  fdung * fyear + fblock, family=Gamma(link="identity"))
res3 <-  glm((Biomass..g.m².) ~  fdung + fyear + fblock, family=Gamma(link="identity"))

I expect the third model to be the most simplistic one and want to confirm this by an information criterion. However, when looking at AIC and BIC I get this output.

AIC(res1,res2,res3)     BIC(result1,res2,res3)
        df      AIC            df      BIC
res1    49 5271.617     res1   49 5465.198
res2    16 5334.234     res2   16 5397.44
res3    10 5331.253     res3   10 5370.760

For AIC, the most complex model is "best" and for BIC the one with fewest df is best. I am thinking that with regard to my aim (identify effects on biomass) I should trust BIC.

Am I wrong here with my conclusion?

I already tried mixed effect models with the fblock as random factor but then the model with the Gamma distribution did not work any more and also I could not use fblock as fixed effect any more (leading to NAs for fblock), but this is not part of my question.

  • $\begingroup$ How many observations do you have? Manual: BIC is defined as AIC(object, ..., k = log(nobs(object))) $\endgroup$ – Max Gordon Nov 17 '11 at 11:08
  • 1
    $\begingroup$ Are result1 and res1 in AIC and BIC different results in your notation? $\endgroup$ – Dmitrij Celov Nov 17 '11 at 14:07

AIC and BIC both penalize models for complexity, but they impose different penalties.

  • AIC = $2k - 2 \log(L)$ where $k$ is number of parameters and $L$ is likelihood.

  • BIC = $k*\log(n) - 2\log(L)$ where $n$ is number of subjects.

So, if $\log(n) > 2$, BIC penalizes more severely; this is pretty much always the case, since $e^2 = 7.3$ and it's rare to have less than 8 subjects.

As to which to use - there's debate about it; see e.g. Burnham and Anderson, 2002 Model Selection and Multimodel Inference (a good book); they also have an article about the comparison in Sociological Methods and Research


My two cents on this topic: if you are really testing hypotheses then I would not use information theoretic approaches, but rather classical hypothesis testing approaches (i.e., I would use drop1 on res1 to do likelihood ratio tests). I personally prefer AIC to BIC in ecology because I think its logic more closely matches what we think about the systems: http://emdbolker.wikidot.com/blog:aic-vs-bic . But if I find myself trying to pick the model that most closely matches a "true" model with some number of non-zero parameters, I would hypothesis-test instead.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.