6
$\begingroup$

When comparing results obtained with different models in R, what should I look for to select the best one?

If I use for example the following 4 models applied to the same presence/absence sample taken from a species dataset, with the same variables:

  • Generalized linear model

  • Generalized additive models Classification

  • Regression Tree

  • Artificial Neural Networks

Should I compare all methods by AIC, Kappa, or cross-validation?

Will I ever be certain of selecting the best model?

What happens if I compare those 4 models prediction with a Bayes factor? Can I compare them?

$\endgroup$
16
$\begingroup$

You are using a wide range of different types of models, and that makes this an interesting situation. Usually, when people say they are engaged in model selection, they mean that they have one type of model, with differing sets of predictors (for example, a multiple regression model with variables A, B, C & D, versus A, B & A*B, etc.). Note that in order to determine the best model, we need to specify what 'best' means; because you are focusing on data mining approaches, I am assuming that you want to maximize predictive accuracy. Let me say a couple of things:

  1. Can you / should you compare them with a Bayes factor? I suspect this can be done, but I have little expertise there, so I should let another CV contributor address that; there are many here who are quite strong on that topic.
  2. Should I compare all methods by AIC? I would not use the AIC in your situation. In general, I think highly of the AIC, but it is not appropriate for every task. There are different versions of the AIC, but in essence, they work the same: The AIC adjusts a goodness-of-fit measure for the ability of a model to produce goodness-of-fit. It does that by penalizing the model for the number of parameters it has. Thus, this assumes that every parameter contributes equally to the ability of a model to fit data. When comparing one multiple regression model to another multiple regression model, that is true. However, it is not at all clear that the addition of another parameter to a multiple regression model equally adds to the ability of the model to fit data as adding another parameter to a very different type of model (e.g., a neural network model, or a classification tree).
  3. Should I compare all methods by Kappa? I also know somewhat less about using Kappa for this goal, but here is a resource with some good general information about it, and here is a paper I stumbled across that does use it in this way, and may be helpful to you (n.b., I haven't read it).
  4. Should I compare all methods by cross-validation? This is probably your best bet. The model selected is the one that minimizes prediction error on a holdout set.
  5. "Will I ever be certain of selecting the best model?" Nope. We're playing a probabilistic game here, and that's just the way it goes, unfortunately. One approach that is probably worth your while is to bootstrap your data, and apply the model selection approach of your choice to each bootsample. This will give you an idea about how clearly one model is favored over the rest. This will be computationally expensive (to say the least), but a small number of iterations should suffice for your purposes, I should think 100 would be enough.
$\endgroup$
  • 4
    $\begingroup$ Optimal classifier selection and negative bias in error rate estimation: An empirical study on high-dimensional prediction is an interesting paper discussing over-optimistic prediction errors when selecting a 'best' model among several ones, including those not reported in the final paper, by AL Boulesteix. (When checking the reference, I came across this internal report that I haven't time to read, Correcting the optimally selected resampling-based error rate: A smooth analytical alternative to nested cross-validation.) $\endgroup$ – chl Feb 15 '12 at 19:19
  • $\begingroup$ @gung Would you then recommend: 1) First draw 100 bootstrap samples from dataset, 2) Find optimal set of predictors for each of the 4 models for each bootstrap sample, 3) Find best-fitting model per each bootstrap, 4) Finally, settle on best model of the four (which may not be as straightforward as choosing model which occupies most #1 slots in the 100 bootstrap samples). $\endgroup$ – RobertF Mar 18 '13 at 16:15
  • 2
    $\begingroup$ @RobertF, something like that might be the way to go. However, I would probably not mix in finding the "optimal set of predictors" w/ the rest. 1st, that adds enormous additional complexity to the problem, & most approaches to doing so are likely to lead to worse models than just fitting the full model (cf, my answer here: algorithms-for-automatic-model-selection). In other words, you would need to do cross validations w/i your cross validations leading to a combinatorial explosion. $\endgroup$ – gung - Reinstate Monica Mar 18 '13 at 16:50
  • $\begingroup$ Kullback Liebler measures of discrepancy, of which the AIC is a nice form, is a powerful and strong parametric measure. That would have been our best bet if likelihoods were comparable. Information theory has something called normalized likelihood. With the use of normalized likelihoods, we might be able to use AIC across model classes. I am currently working on this topic. $\endgroup$ – Chamberlain Foncha Nov 14 '14 at 9:36
7
$\begingroup$

In my mind, cross-validation is a pretty solid gold standard for making comparisons that focus on models' abilities to predict new data. That said, for the GLM case at least, AIC has been demonstrated (Stone, 1977) to be asymptotically equivalent to cross-validation, so if you're ok with asymptotic assumption, you can save yourself some compute time by going with AIC rather than computing the full cross-validation.

$\endgroup$
5
$\begingroup$

Assuming yo are using classification error or something similar as your performance measure, then why don't you try cross validation of all models?

Split your data into, say, 10 chunks, and then do 10 rounds of build and test using one of those chunks as the test set and the other nine as the training (ie. round1: train 2-10, test 1. round2: train 1+3-10, test 2. round3: train 1-2+4-10, test 3).

This approach helps you find which algorithm (and which parameters for those models), perform the best.

One of the things I struggled with at first, was that it is not so much the actual model that gets built that matters, so much as the function you call and the parameters you provide to it that are important.

$\endgroup$
  • 6
    $\begingroup$ There are books written on this topic. The one I use is Harrell's "Regression Modeling Strategies." He criticizes N-fold cross-validation as an inefficient use of data, and suggests (as well as offering the software to implement) that you use bootstrapped methods. $\endgroup$ – DWin Feb 15 '12 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.