Is cross-validation the most important measure of a predictive model's effectiveness? Why bother with p-values, R squared, etc. ... Model size is not a factor with the computing power available now so why not just run multiple iterations of all possible sets of input variables and see which one has lowest cross-validation error?.
 A: I would personally favor cross-validated score evaluation because:


*

*it is easily interpretable by the analyst provided that the underlying score function (accuracy, f1-score, RMSE...) is interpretable too,

*it gives an idea of the uncertainty by looking at the stdev of the score values across CV folds,

*it gives a way to decompose the error into bias (error measured on train folds) and variance (difference of errors measured on train and  test folds).

Model size is not a factor with the computing power 

This is not always true: deep learning machine learning models for instance have a model size that is often limited by the hardware (typically the amount of RAM on the GPU card).
A: Two scenarios spring to mind where you wouldn't want to just run iterations of all possible models:


*

*Your model is in a clinical setting. For example, a nurse takes some measurements and uses it to predict something. If you include every possible covariate, then you are more likely to get missing values. Especially if some of the covariates require sensitive information.

*In your model, $p>> n$, however, your covariates can be "grouped". For example, one set of covariates could be "lifestyle", another could be socio-economic class. It might be better to try and select a few variables from each group.
A: A good reason not to do this is that the cross-validation estimator has a finite variance, so if you evaluate it on many choices of input variables you will end up with a set that explains the data you have well, but will generalise poorly as it has effectively learned the noise that is particular to that dataset. The more choices you investigate, the worse the problem gets. Often you end up with a worse predictor than a regularised model, with all the features, such as ridge regression.  So if you are interested in predictive performance, don't perform feature selection at all, instead use regularisation.  This is the advice given in Millar's monograph on subset selection in regression, and in my experience, he is right.
