Choosing the best model from among different "best" models

Question

How do you choose a model from among different models chosen by different methods (e.g. backwards or forwards selection)?

Also what is a parsimonious model?

Answer 1

A parsimonious model is a model that accomplishes a desired level of explanation or prediction with as few predictor variables as possible.

For model evaluation there are different methods depending on what you want to know. There are generally two ways of evaluating a model: Based on predictions and based on goodness of fit on the current data. In the first case you want to know if your model adequately predicts new data, in the second you want to know whether your model adequatelly describes the relations in your current data. Those are two different things.

Evaluating based on predictions

The best way to evaluate models used for prediction, is crossvalidation. Very briefly, you cut your dataset in eg. 10 different pieces, use 9 of them to build the model and predict the outcomes for the tenth dataset. A simple mean squared difference between the observed and predicted values give you a measure for the prediction accuracy. As you repeat this ten times, you calculate the mean squared difference over all ten iterations to come to a general value with a standard deviation. This allows you again to compare two models on their prediction accuracy using standard statistical techniques (t-test or ANOVA).

A variant on the theme is the PRESS criterion (Prediction Sum of Squares), defined as

$\displaystyle\sum^{n}_{i=1} \left(Y_i - \hat{Y}_{i(-i)}\right)^2$

Where $\hat{Y}_{i(-i)}$ is the predicted value for the ith observation using a model based on all observations minus the ith value. This criterion is especially useful if you don't have much data. In that case, splitting your data like in the crossvalidation approach might result in subsets of data that are too small for a stable fitting.

Evaluating based on goodness of fit

Let me first state that this really differs depending on the model framework you use. For example, a likelihood-ratio test can work for Generalized Additive Mixed Models when using the classic gaussian for the errors, but is meaningless in the case of the binomial variant.

First you have the more intuitive methods of comparing models. You can use the Aikake Information Criterion (AIC) or the Bayesian Information Criterion (BIC) to compare the goodness of fit for two models. But nothing tells you that both models really differ.

Another one is the Mallow's Cp criterion. This essentially checks for possible bias in your model, by comparing the model with all possible submodels (or a careful selection of them). See also http://www.public.iastate.edu/~mervyn/stat401/Other/mallows.pdf

If the models you want to compare are nested models (i.e. all predictors and interactions of the more parsimonious model occur also in the more complete model), you can use a formal comparison in the form of a likelihood ratio test (or a Chi-squared or an F test in the appropriate cases, eg when comparing simple linear models fitted using least squares). This test essentially controls whether the extra predictors or interactions really improve the model. This criterion is often used in forward or backward stepwise methods.

About automatic model selection

You have advocates and you have enemies of this method. I personally am not in favor of automatic model selection, especially not when it's about describing models, and this for a number of reasons:

• In every model you should have checked that you deal adequately with confounding. In fact, many datasets have variables that should never be put in a model at the same time. Often people forget to control for that.
• Automatic model selection is a method to create hypotheses, not to test them. All inference based on models originating from Automatic model selection is invalid. No way to change that.
• I've seen many cases where starting at a different starting point, a stepwise selection returned a completely different model. These methods are far from stable.
• It's also difficult to incorporate a decent rule, as the statistical tests to compare two models require the models to be nested. If you use eg AIC, BIC or PRESS, the cutoff for when a difference is really important is arbitrary chosen.

So basically, I see more in comparing a select set of models chosen beforehand. If you don't care about statistical evaluation of the model and hypothesis testing, you can use crossvalidation to compare the predictive accuracy of your models.

But if you're really after variable selection for predictive purposes, you might want to take a look to other methods for variable selection, like Support Vector Machines, Neural Networks, Random Forests and the likes. These are far more often used in eg medicine to find out which of the thousand measured proteins can adequately predict whether you have cancer or not. Just to give a (famous) example :

• https://www.nature.com/articles/nm0601_673
• https://doi.org/10.1023/A:1012487302797

All these methods have regression variants for continuous data as well.

Answer 2

Parsimony is your enemy. Nature does not act parsimoneously, and datasets do not have enough information to allow one to choose the "right" variables. It doesn't matter very much which method you use or which index you use as a stopping rule. Variable selection without shrinkage is almost doomed. However limited backwards stepdown (with $\alpha=0.50$) can sometimes be helpful. It works simply because it will not delete many variables.

Answer 3

Using backwards or forwards selection is a common strategy, but not one I can recommend. The results from such model building are all wrong. The p-values are too low, the coefficients are biased away from 0, and there are other related problems.

If you must do automatic variable selection, I would recommend using a more modern method, such as LASSO or LAR.

I wrote a SAS presentation on this, entitled "Stopping Stepwise: Why Stepwise and Similar Methods are Bad and what you should Use"

But, if possible, I'd avoid these automated methods altogether, and rely on subject matter expertise. One idea is to generate 10 or so reasonable models, and compare them based on an information criterion. @Nick Sabbe listed several of these in his response.

Answer 4

The answer to this will greatly depend upon your goal. You may be looking for statistically significant coefficients, or you may be out to avoid as many missclassifications as possible when predicting the outcome for new observations, or you may simply be interested in the model with the least false positives; perhaps you simply want the curve that is 'closest' to the data.

In any of the cases above, you need some sort of measure for what you are looking for. Some popular measures with different applications are AUC, BIC, AIC, residual error,...

You calculate the measure that best matches your goal for each model, and then compare the 'scores' for each model. This leads to the best model for your goal.

Some of these measures (e.g. AIC) place an extra stress on the number of nonzero coefficients in the model, because using too many could be simply overfitting the data (so that the model is useless if you use it for new data, let alone for the population). There may be other reasons for requiring a model to hold 'as little as possible' variables, e.g. if it is simply costly to measure all of them for prediction. The 'simplicity of' or 'small number of variables in' a model is typically refered to as its parsimony.

So in short, a parsimoneous model is a 'simple' model, not holding too many variables.

As often with these type of questions, I will refer you to the excellent book Elements of Statistical Learning for deeper information on the subject and related issues.