# Choosing the best model from among different “best” models

How do you choose a model from among different models chosen by different methods (e.g. backwards or forwards selection)?

Also what is a parsimonious model?

A parsimonious model is a model that accomplishes a desired level of explanation or prediction with as few predictor variables as possible.

For model evaluation there are different methods depending on what you want to know. There are generally two ways of evaluating a model: Based on predictions and based on goodness of fit on the current data. In the first case you want to know if your model adequately predicts new data, in the second you want to know whether your model adequatelly describes the relations in your current data. Those are two different things.

## Evaluating based on predictions

The best way to evaluate models used for prediction, is crossvalidation. Very briefly, you cut your dataset in eg. 10 different pieces, use 9 of them to build the model and predict the outcomes for the tenth dataset. A simple mean squared difference between the observed and predicted values give you a measure for the prediction accuracy. As you repeat this ten times, you calculate the mean squared difference over all ten iterations to come to a general value with a standard deviation. This allows you again to compare two models on their prediction accuracy using standard statistical techniques (t-test or ANOVA).

A variant on the theme is the PRESS criterion (Prediction Sum of Squares), defined as

$\displaystyle\sum^{n}_{i=1} \left(Y_i - \hat{Y}_{i(-i)}\right)^2$

Where $\hat{Y}_{i(-i)}$ is the predicted value for the ith observation using a model based on all observations minus the ith value. This criterion is especially useful if you don't have much data. In that case, splitting your data like in the crossvalidation approach might result in subsets of data that are too small for a stable fitting.

## Evaluating based on goodness of fit

Let me first state that this really differs depending on the model framework you use. For example, a likelihood-ratio test can work for Generalized Additive Mixed Models when using the classic gaussian for the errors, but is meaningless in the case of the binomial variant.

First you have the more intuitive methods of comparing models. You can use the Aikake Information Criterion (AIC) or the Bayesian Information Criterion (BIC) to compare the goodness of fit for two models. But nothing tells you that both models really differ.

Another one is the Mallow's Cp criterion. This essentially checks for possible bias in your model, by comparing the model with all possible submodels (or a careful selection of them). See also http://www.public.iastate.edu/~mervyn/stat401/Other/mallows.pdf

If the models you want to compare are nested models (i.e. all predictors and interactions of the more parsimonious model occur also in the more complete model), you can use a formal comparison in the form of a likelihood ratio test (or a Chi-squared or an F test in the appropriate cases, eg when comparing simple linear models fitted using least squares). This test essentially controls whether the extra predictors or interactions really improve the model. This criterion is often used in forward or backward stepwise methods.

You have advocates and you have enemies of this method. I personally am not in favor of automatic model selection, especially not when it's about describing models, and this for a number of reasons:

• In every model you should have checked that you deal adequately with confounding. In fact, many datasets have variables that should never be put in a model at the same time. Often people forget to control for that.
• Automatic model selection is a method to create hypotheses, not to test them. All inference based on models originating from Automatic model selection is invalid. No way to change that.
• I've seen many cases where starting at a different starting point, a stepwise selection returned a completely different model. These methods are far from stable.
• It's also difficult to incorporate a decent rule, as the statistical tests to compare two models require the models to be nested. If you use eg AIC, BIC or PRESS, the cutoff for when a difference is really important is arbitrary chosen.

So basically, I see more in comparing a select set of models chosen beforehand. If you don't care about statistical evaluation of the model and hypothesis testing, you can use crossvalidation to compare the predictive accuracy of your models.

But if you're really after variable selection for predictive purposes, you might want to take a look to other methods for variable selection, like Support Vector Machines, Neural Networks, Random Forests and the likes. These are far more often used in eg medicine to find out which of the thousand measured proteins can adequately predict whether you have cancer or not. Just to give a (famous) example :

http://www.nature.com/nm/journal/v7/n6/abs/nm0601_673.html

All these methods have regression variants for continuous data as well.

• Which model would you select between Mallows Cp and backward selection? Also are models with low SSE and significant coefficients good? – tom Oct 26 '11 at 14:05
• @tom : you're comparing apples with oranges. backward selection is a method, Mallows Cp is a criterion. Mallow's Cp can be used as a criterion for backwards selection. And as you can read, I don't do backward selection. If I need to select variables, I use appropriate methods for that. I didn't mention the LASSO and LAR methods Peter Flom referred to, but they're definitely worth a try too. – Joris Meys Oct 26 '11 at 14:08
• @FrankHarrell a little simulation can prove that the correlation between the p-values (presuming you're talking about the F-test or equivalent) and the AIC is nonexistent (0.01 in my simulation). So no, there's no relation between the P-values and the AIC. Same for BIC and Cp. Another little simulation will also prove that one gets pretty different results in a stepwise procedure depending on the criterium you use. So no: Cp, AIC, BIC are in no way just transformations of P-values. In fact, if looking at the formulas I can in no way point to a mathematical link or transformation. – Joris Meys Oct 27 '11 at 11:10
• @FrankHarrell which doesn't mean that I'm advocating pro stepwise, in contrary. But your statement is at least formulated a bit strong. – Joris Meys Oct 27 '11 at 11:12
• I believe that is incorrect Joris. You should be able to solve for the transformations that relate AIC, BIC, Cp, and P-values from partial tests. And if you think AIC, BIC, and Cp relieve you of multiplicity problems, think again. – Frank Harrell Oct 27 '11 at 23:01

Parsimony is your enemy. Nature does not act parsimoneously, and datasets do not have enough information to allow one to choose the "right" variables. It doesn't matter very much which method you use or which index you use as a stopping rule. Variable selection without shrinkage is almost doomed. However limited backwards stepdown (with $\alpha=0.50$) can sometimes be helpful. It works simply because it will not delete many variables.

• The question is not about stepwise, but about selecting the best model among the results of different approaches... – Joris Meys Oct 26 '11 at 13:38
• I very much like "parsimony is your enemy". – Peter Flom Oct 26 '11 at 13:51
• Thanks Peter. Joris - selecting from among different approaches differs a bit from stepwise selection, but not much. – Frank Harrell Oct 26 '11 at 17:02

Using backwards or forwards selection is a common strategy, but not one I can recommend. The results from such model building are all wrong. The p-values are too low, the coefficients are biased away from 0, and there are other related problems.

If you must do automatic variable selection, I would recommend using a more modern method, such as LASSO or LAR.

I wrote a SAS presentation on this, entitled "Stopping Stepwise: Why Stepwise and Similar Methods are Bad and what you should Use"

But, if possible, I'd avoid these automated methods altogether, and rely on subject matter expertise. One idea is to generate 10 or so reasonable models, and compare them based on an information criterion. @Nick Sabbe listed several of these in his response.

• +1 for the article reference. Although I do not code in SAS, I read it several months ago and found it to be a nice, high level treatment of the issue. – Josh Hemann Oct 26 '11 at 19:09

The answer to this will greatly depend upon your goal. You may be looking for statistically significant coefficients, or you may be out to avoid as many missclassifications as possible when predicting the outcome for new observations, or you may simply be interested in the model with the least false positives; perhaps you simply want the curve that is 'closest' to the data.

In any of the cases above, you need some sort of measure for what you are looking for. Some popular measures with different applications are AUC, BIC, AIC, residual error,...

You calculate the measure that best matches your goal for each model, and then compare the 'scores' for each model. This leads to the best model for your goal.

Some of these measures (e.g. AIC) place an extra stress on the number of nonzero coefficients in the model, because using too many could be simply overfitting the data (so that the model is useless if you use it for new data, let alone for the population). There may be other reasons for requiring a model to hold 'as little as possible' variables, e.g. if it is simply costly to measure all of them for prediction. The 'simplicity of' or 'small number of variables in' a model is typically refered to as its parsimony.

So in short, a parsimoneous model is a 'simple' model, not holding too many variables.

As often with these type of questions, I will refer you to the excellent book Elements of Statistical Learning for deeper information on the subject and related issues.

• Nice book you recommend there. Another one I could recommend is Applied Linear Statistical Models which contains a few sections on selection criteria, model selection and model comparison. – Joris Meys Oct 26 '11 at 12:15