Model evaluation and comparison for selecting the best model

Question

When comparing results obtained with different models in R, what should I look for to select the best one?

If I use for example the following 4 models applied to the same presence/absence sample taken from a species dataset, with the same variables:

• Generalized linear model

• Generalized additive models Classification

• Regression Tree

• Artificial Neural Networks

Should I compare all methods by AIC, Kappa, or cross-validation?

Will I ever be certain of selecting the best model?

What happens if I compare those 4 models prediction with a Bayes factor? Can I compare them?

Answer 1

You are using a wide range of different types of models, and that makes this an interesting situation. Usually, when people say they are engaged in model selection, they mean that they have one type of model, with differing sets of predictors (for example, a multiple regression model with variables A, B, C & D, versus A, B & A*B, etc.). Note that in order to determine the best model, we need to specify what 'best' means; because you are focusing on data mining approaches, I am assuming that you want to maximize predictive accuracy. Let me say a couple of things:

1. Can you / should you compare them with a Bayes factor? I suspect this can be done, but I have little expertise there, so I should let another CV contributor address that; there are many here who are quite strong on that topic.
2. Should I compare all methods by AIC? I would not use the AIC in your situation. In general, I think highly of the AIC, but it is not appropriate for every task. There are different versions of the AIC, but in essence, they work the same: The AIC adjusts a goodness-of-fit measure for the ability of a model to produce goodness-of-fit. It does that by penalizing the model for the number of parameters it has. Thus, this assumes that every parameter contributes equally to the ability of a model to fit data. When comparing one multiple regression model to another multiple regression model, that is true. However, it is not at all clear that the addition of another parameter to a multiple regression model equally adds to the ability of the model to fit data as adding another parameter to a very different type of model (e.g., a neural network model, or a classification tree).
3. Should I compare all methods by Kappa? I also know somewhat less about using Kappa for this goal, but here is a resource with some good general information about it, and here is a paper I stumbled across that does use it in this way, and may be helpful to you (n.b., I haven't read it).
4. Should I compare all methods by cross-validation? This is probably your best bet. The model selected is the one that minimizes prediction error on a holdout set.
5. "Will I ever be certain of selecting the best model?" Nope. We're playing a probabilistic game here, and that's just the way it goes, unfortunately. One approach that is probably worth your while is to bootstrap your data, and apply the model selection approach of your choice to each bootsample. This will give you an idea about how clearly one model is favored over the rest. This will be computationally expensive (to say the least), but a small number of iterations should suffice for your purposes, I should think 100 would be enough.

Answer 2

In my mind, cross-validation is a pretty solid gold standard for making comparisons that focus on models' abilities to predict new data. That said, for the GLM case at least, AIC has been demonstrated (Stone, 1977) to be asymptotically equivalent to cross-validation, so if you're ok with asymptotic assumption, you can save yourself some compute time by going with AIC rather than computing the full cross-validation.

Answer 3

Assuming yo are using classification error or something similar as your performance measure, then why don't you try cross validation of all models?

Split your data into, say, 10 chunks, and then do 10 rounds of build and test using one of those chunks as the test set and the other nine as the training (ie. round1: train 2-10, test 1. round2: train 1+3-10, test 2. round3: train 1-2+4-10, test 3).

This approach helps you find which algorithm (and which parameters for those models), perform the best.

One of the things I struggled with at first, was that it is not so much the actual model that gets built that matters, so much as the function you call and the parameters you provide to it that are important.