I am starting an analysis, for which I have a binomial response variable (species relative abundance) and continuous predictors (habitat variables). I have done some data exploration, and there is reason to expect some non-linear relationships and spatial autocorrelation. I am starting with exploratory GAMs, and plan to use GLMs but will likely need to try GLMM to account for random effects (year, month, site). I want to follow the Burnham and Anderson (2002) approach to model selection, based on an a priori set of models. I know enough about the system in order to make a set of models (which variables to include, which interactions to include), however I do not know which types of models will be "best" (most parsimonious).

So my question is: Since I can not compare between different types of models (GAM, GLM, GLMM) using this approach, and the approach advocates against stepwise selection and unnecessary data-dredging, at what point is it acceptable to play around with variations of one model? Can I try several variations on my global model, using different methods to account for autocorrelation, random effects, overdispersion, etc, and then use the best as a basis for my subset of models? Or should I just choose what I think will be the best, and then after model selection can try different options on the best model (or group of models)?



1 Answer 1


I think that cross-validation is the best choice in your case. The reason is that your goal, ultimately, is to make predictions about the population. So you want to choose a model accordingly. Since you do not know the correct model, the convergence guarantees for AIC (and BIC) do not apply. However, there is a proof that shows that the cross validation solution is close to the structural risk minimization solution for model selection (you can find the proof in the appendix here).

So just to review, the structural risk minimization solution (SRM solution) selects a model that has the lowest generalization error. You never actually have access to the generalization error, but theoretically (I mean with math proofs) you can bound it by the empirical error (the error of your model on your data), a complexity term, and a term that's related to the variance of your distribution. The optimal solution with these bounds coincide with the SRM solution, which you can approximate with cross-validation. Thus, cross-validation has a strong theoretical justification and allows you to find the best solution for a class of models (even if the true model is not included).

Also, a nice thing here is that you can also play with the different parameters with cross-validation using grid-search for each model and compare across different models. Just make sure you do cross validation correctly! (I see people screw this up all the time).

You can find the n-fold cross-validation procedure (which is used in the proof) on slide 17 of that pdf.

Grid Search: The basic idea of grid search is to do cross validation with a manually chosen set of parameters. So for example, you could compare GLMs that are quadratic, cubic, etc. This should cover a wide range. Usually you should plot the cross-validation performance as a function of the parameter choice and just choose the performance with the best cross-validation performance (lowest CV error). For a concrete example, check out my tutorial on SVMs. Scroll all the way down to In[29] to see the code. In that example I am using grid search over an SVM parameter C. You can see the sort of plot I'm talking about in Out[30].

Make sense?

  • $\begingroup$ Thanks for the answer. I have not done cross-validation, but looked into it and it does seem that it will allow me to compare between non-nested models (GLM, GAM, GAMM, etc) by comparing error. I don't understand the last part about grid-search, but I need to do more reading on cross-validation. So I guess the workflow would be: Data exploration, a priori model set (but with various types of models), model selection through cross-validation, and then basic model validation on best model (residuals etc)? $\endgroup$
    – snowtosurf
    Commented Jun 9, 2015 at 23:12
  • $\begingroup$ Close, you should do cross validation to compare all models… that includes changing the parameters. So, you could do cross validation with different parameters for each model class (so GLM then GAM then GAMM). Then once you have the best model from each model class (via parameters) you can compare the cross validation performance for each of the model classes. I'm updating my post above to explain grid search. $\endgroup$ Commented Jun 10, 2015 at 0:15
  • $\begingroup$ @justanotherbrain, the second link (the one towards your SVM tutorial) is dead. Also, in slide 17 of your first link (Michael Rebady's presentation), he splits his data several times but that's only possible when you have large datasets, right? In ecology (which seems to be OP's field and mine), we rarely have many observations. Would that work with n < 100? $\endgroup$
    – Fanfoué
    Commented Apr 19, 2022 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.