This post has been heavily edited from its previous version (three months ago).
I am investigating habitat selection of 35 territorial wolves over several years of denning seasons (41 wolf-years total); each wolf has ~ 195 locations per year. I am comparing wolf locations within a territory (used = 1) to random locations (used = 0, ~ 2800 locations), also within the wolf’s territory. The distance from each used or random point was measured to certain habitat features, such as water, roads, and habitat types. We are interested in whether wolves with a greater density of habitat features in their territory use that feature more or less. I'm using a binomial GLMM.
The recommended strategy (Bolker et al., 2009; Zuur et al., 2009) for determining the appropriate top model seems to be to first determine the correct random effects structure using a feasible level of complexity in the fixed effects and then determine the correct fixed effects structure.
Example model (some terms are multiplied by their availability, since not all wolves have every habitat feature available to them within their territory).
use ~ fixed effects: Water + Main_Roads*availability + Tertiary_Roads*Availability + Conifer*availability + Deciduous*availability + Lowland*availability + OldCuts*availability + random effects: (Water + Main_Roads*availability + Tertiary_Roads*Availability + Conifer*availability + Deciduous*availability + Lowland*availability + OldCuts*availability / subject = year(wolf))
We originally planned to use the AIC to choose the “best” random effects model structure first; but we are running into some strange behaviour. Models with increasingly complex random effects structures result in lower (better) AIC values, but often, the random effects in these more complex models are not significant. Models that are similar in structure, but that do not contain the non-significant random effect have higher AIC values. How is this possible? The more complex model has more uninformative parameters, yet has a lower AIC. My AIC values range from 53511-53629 for a group of 25 candidate models with decreasingly complex random effects structures. Perhaps these large values, with such (relatively) small differences among them, indicate poor model fit overall and a lack of ability to discern meaningfully among them?
I read this post, Using AIC, for model selection when both models are equally weighted, and one model has fewer parameters and also the suggested paper, Arnold (2010) - it discusses that models within 2 ∆AIC values of a top-supported model with uninformative parameters should not be considered “competitive”, and can be ignored, but even my top model is not “competitive” in these terms, as it includes multiple uninformative random effects. Is it possible that multiplying each parameter by its availability is affecting the calculation of AIC (because there are many more parameters) and that a difference of >> 2 AIC units should not be considered “competitive” in my case?
My main questions are:
Is AIC an inappropriate measure to use for choosing between random effects structures in this case? Or would it be appropriate to choose the “top random effect structure model” based on which random slope parameters are significant, and not use the AIC? Or should I be concerned that this behaviour indicates that something is very wrong with the underlying data or test? I am using SAS PROC GLIMMIX to analyze these data (as I’d like to compute sandwich standard errors, and as of yet this isn't possible in R).
Most posts on CrossValidated that discuss this question regard fixed effects, not random effects. The post Should covariates that are not statistically significant be 'kept in' when creating a model? suggests it may be dangerous to discard models with uninformative parameters, but I'm not sure the advice is appropriate for my situation.
The post The best model, according to both AIC and BIC, contains only a non significant term also discusses this, but the OP chose to solve their problem by using ML instead of REML, which affected which model was chosen as "best". The general advice is to use REML to choose between random effects structures and ML for fixed effects structures for lmmms... but SAS will not (to my knowledge) output AIC values when I use REML, and Doug Bates once said that REML estimates are not well defined for GLMMs. So I have been using the Laplace approximation (and ML) so far... (I plan to switch to Gauss-Hermite quadrature in my final model selection step for fixed effects).