Proper use of model inference (AIC) (Burnham and Anderson) - when to explore more models

Question

I am starting an analysis, for which I have a binomial response variable (species relative abundance) and continuous predictors (habitat variables). I have done some data exploration, and there is reason to expect some non-linear relationships and spatial autocorrelation. I am starting with exploratory GAMs, and plan to use GLMs but will likely need to try GLMM to account for random effects (year, month, site). I want to follow the Burnham and Anderson (2002) approach to model selection, based on an a priori set of models. I know enough about the system in order to make a set of models (which variables to include, which interactions to include), however I do not know which types of models will be "best" (most parsimonious).

So my question is: Since I can not compare between different types of models (GAM, GLM, GLMM) using this approach, and the approach advocates against stepwise selection and unnecessary data-dredging, at what point is it acceptable to play around with variations of one model? Can I try several variations on my global model, using different methods to account for autocorrelation, random effects, overdispersion, etc, and then use the best as a basis for my subset of models? Or should I just choose what I think will be the best, and then after model selection can try different options on the best model (or group of models)?

Thanks

Answer 1

I think that cross-validation is the best choice in your case. The reason is that your goal, ultimately, is to make predictions about the population. So you want to choose a model accordingly. Since you do not know the correct model, the convergence guarantees for AIC (and BIC) do not apply. However, there is a proof that shows that the cross validation solution is close to the structural risk minimization solution for model selection (you can find the proof in the appendix here).

So just to review, the structural risk minimization solution (SRM solution) selects a model that has the lowest generalization error. You never actually have access to the generalization error, but theoretically (I mean with math proofs) you can bound it by the empirical error (the error of your model on your data), a complexity term, and a term that's related to the variance of your distribution. The optimal solution with these bounds coincide with the SRM solution, which you can approximate with cross-validation. Thus, cross-validation has a strong theoretical justification and allows you to find the best solution for a class of models (even if the true model is not included).

Also, a nice thing here is that you can also play with the different parameters with cross-validation using grid-search for each model and compare across different models. Just make sure you do cross validation correctly! (I see people screw this up all the time).

You can find the n-fold cross-validation procedure (which is used in the proof) on slide 17 of that pdf.

Grid Search: The basic idea of grid search is to do cross validation with a manually chosen set of parameters. So for example, you could compare GLMs that are quadratic, cubic, etc. This should cover a wide range. Usually you should plot the cross-validation performance as a function of the parameter choice and just choose the performance with the best cross-validation performance (lowest CV error). For a concrete example, check out my tutorial on SVMs. Scroll all the way down to In[29] to see the code. In that example I am using grid search over an SVM parameter C. You can see the sort of plot I'm talking about in Out[30].

Make sense?