# Is cross-validation the most important measure of a predictive model's effectiveness?

Why bother with p-values, R squared, etc. ... Model size is not a factor with the computing power available now so why not just run multiple iterations of all possible sets of input variables and see which one has lowest cross-validation error?.

• You must distinguish between two things -- model selection (which type/parameters of the model is best) and model evaluation (what accuracy the model should have on new data). If the first part is present, you must either remove it (with regularization or similar idea) or include it in model assessment CV or you'll get overoptimistic results.
– user88
Feb 27 '12 at 12:03

A good reason not to do this is that the cross-validation estimator has a finite variance, so if you evaluate it on many choices of input variables you will end up with a set that explains the data you have well, but will generalise poorly as it has effectively learned the noise that is particular to that dataset. The more choices you investigate, the worse the problem gets. Often you end up with a worse predictor than a regularised model, with all the features, such as ridge regression. So if you are interested in predictive performance, don't perform feature selection at all, instead use regularisation. This is the advice given in Millar's monograph on subset selection in regression, and in my experience, he is right.

I would personally favor cross-validated score evaluation because:

• it is easily interpretable by the analyst provided that the underlying score function (accuracy, f1-score, RMSE...) is interpretable too,

• it gives an idea of the uncertainty by looking at the stdev of the score values across CV folds,

• it gives a way to decompose the error into bias (error measured on train folds) and variance (difference of errors measured on train and test folds).

Model size is not a factor with the computing power

This is not always true: deep learning machine learning models for instance have a model size that is often limited by the hardware (typically the amount of RAM on the GPU card).

Two scenarios spring to mind where you wouldn't want to just run iterations of all possible models:

1. Your model is in a clinical setting. For example, a nurse takes some measurements and uses it to predict something. If you include every possible covariate, then you are more likely to get missing values. Especially if some of the covariates require sensitive information.

2. In your model, $p>> n$, however, your covariates can be "grouped". For example, one set of covariates could be "lifestyle", another could be socio-economic class. It might be better to try and select a few variables from each group.

• +1, though you haven't spelled out why this last is better than using all possible subsets of predictors. Feb 27 '12 at 23:02
• You didn't spell out why all possible subsets would ever be a good idea. Feb 27 '14 at 12:45