My analysis uses a negative binomial GLMM with total revisits as the dependent variable, treatment (factor with 4 levels: 0ppb, 4.8ppb, 20ppb, 133ppb) and size as fixed effects and colony as a random effect. I am using model selection to determine the best model or set of models, and want to use AIC corrected for small sample sizes (AICc) as my n/K < 40 (n=56). I am using R with the glmmADMB and MuMIn packages.

My problem is that it seems that AICc is penalising model complexity to harshly. Below is the summary output of the full model, and the model selection table using standard AIC and that using AICc.

output summary and model selection tables

The AIC puts the model with the interaction term as the best model (although <2 away from size only model), which makes sense as there is a strong significant interaction between treatment and size when comparing the 133ppb treatment level to the control baseline. The order of the models using AIC generally seems to make sense.

The AICc on the other hand seems to simply order them almost by model complexity, giving the model with the interaction the highest AICc. This suggests I should remove the interaction, but this doesn't make sense as it is significant.

The treatment levels 4.8ppb and 20ppb are not significantly different from one another, so I ran a separate analysis with these levels pooled ("FR"). This gave an order of AICc values much closer to the AIC, and much more in line with what I expected from the significance of the parameters in the full model (see image below). I think I can see why this is the case: the model has fewer degrees of freedom when two of the factors are pooled so the penalty for complexity is lower.

enter image description here

However, it is important that for the final analysis I keep these treatment levels separate. I want to continue using the AICc as this is appropriate for my small sample size, but cannot report the model selection output as it stands as it clearly does not reflect the data e.g. there is a strong interaction between treatment and size, but the model including this comes out as having the highest AICc.

It seems to me that the AICc is penalising model complexity too highly at the cost of removing important parameters. I also have three other dependent variables for which the same situation is occurring.

Does anyone have any suggestions for dealing with this?

  • $\begingroup$ Are you concerned about the absolute best AICc to meet a publication requirement? If not, if you predict the same value from each of these models, I am curious how much, if any, the prediction would meaningfully change. $\endgroup$ – jknowles Aug 25 '15 at 13:51
  • $\begingroup$ Thanks for your comment. No, it is not to meet a publication requirement. Are you suggesting that model selection is not appropriate in this case because the difference between the models is small? If so, do you suggest simply using the full model and making predictions from that? The other three dependent variables do not have significant interactions between treatment and size, but have a similar problem with AICc. $\endgroup$ – LizS Aug 25 '15 at 14:54
  • $\begingroup$ I'm not familiar with your data, but it seems to me like the AICc differences are pretty small between models. In my experience GLMM model fit statistics can show changes that do not result in meaningful changes in the actual fitted values or predicted values from the model. I would take an example case, or a set of example cases, and get the fitted values from each model and see how far apart they are for different cases. If they are all really close, there is no reason not to use the model for which you have theoretical support. $\endgroup$ – jknowles Aug 25 '15 at 15:42
  • $\begingroup$ The AICc <...> suggests I should remove the interaction, but this doesn't make sense as it is significant. If statistical significance was the best criterion for including or excluding a regressor, why would you look at AICc? As far as I understand, AICc should be used to determine which model to choose given the limited amount of data (uncer certain assumptions and a certain loss function). If I were you, I would trust AICc more than statistical significance. But I have to admit that I have little experience in model selection... $\endgroup$ – Richard Hardy Sep 4 '15 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.