When are Shao's results on leave-one-out cross-validation applicable?

Question

In his paper Linear Model Selection by Cross-Validation, Jun Shao shows that for the problem of variable selection in multivariate linear regression, the method of leave-one-out cross validation (LOOCV) is 'asymptotically inconsistent'. In plain English, it tends to select models with too many variables. In a simulation study, Shao shows that even for as few as 40 observations, LOOCV can underperform other cross-validation techniques.

This paper is somewhat controversial, and somewhat ignored (10 years after its publication, my chemometrics colleagues had never heard of it and were happily using LOOCV for variable selection...). There is also a belief (I am guilty of this), that its results extend somewhat beyond the original limited scope.

The question, then: how far do these results extend? Are they applicable to the following problems?

1. Variable selection for logistic regression/GLM?
2. Variable selection for Fisher LDA classification?
3. Variable selection using SVM with finite (or infinite) kernel space?
4. Comparison of models in classification, say SVM using different kernels?
5. Comparison of models in linear regression, say comparing MLR to Ridge Regression?
6. etc.

Answer 1

You need to specify the purpose of the model before you can say whether Shao's results are applicable. For example, if the purpose is prediction, then LOOCV makes good sense and the inconsistency of variable selection is not a problem. On the other hand, if the purpose is to identify the important variables and explain how they affect the response variable, then Shao's results are obviously important and LOOCV is not appropriate.

The AIC is asymptotically LOOCV and BIC is asymptotically equivalent to a leave-$v$-out CV where $v=n[1-1/(\log(n)-1)]$ --- the BIC result for linear models only. So the BIC gives consistent model selection. Therefore a short-hand summary of Shao's result is that AIC is useful for prediction but BIC is useful for explanation.

Answer 2

This paper is somewhat controversial, and somewhat ignored

Not really, it's well regarded where the theory of model selection is concerned, though it's certainly misinterpreted. The real issue is how relevant it is to the practice of modeling in the wild. Suppose you perform the simulations for the cases you propose to investigate and determine that LOOCV is indeed inconsistent. The only reason you'd get that is because you already knew the "true" model and could hence determine that the probability of recovering the "true" model does not converge to 1. For modeling in the wild, how often is this true (that the phenomena are described by linear models and the "true" model is a subset of those in consideration)?

Shao's paper is certainly interesting for advancing the theoretical framework. It even provides some clarity: if the "true" model is indeed under consideration, then we have the consistency results to hang our hats on. But I'm not sure how interesting actual simulations for the cases you describe would be. This is largely why most books like EOSL don't focus as much on Shao's result, but instead on prediction/generalization error as a criterion for model selection.

EDIT: The very short answer to your question is: Shao's results are applicable when you're performing least squares estimation, quadratic loss function. Not any wider. (I think there was an interesting paper by Yang (2005?) which investigated whether you could have consistency and efficiency, with a negative answer.)

Answer 3

I would say: everywhere, but I haven't seen a strict proof of it. The intuition behind is such that when doing CV one must hold a balance between train large enough to build sensible model and test large enough so it would be a sensible benchmark.
When dealing with thousands of pretty homogeneous objects, picking one is connected with risk that it is pretty similar to other object that was left in the set -- and then the results would be too optimistic.
On the other hand, in case of few objects there will be no vital difference between LOO and k-fold; $10/10$ is just $1$ and we can't do anything with it.

Answer 4

1) The answer by @ars mentions Yang (2005), "Can The Strengths of AIC and BIC Be Shared?". Loosely speaking, it seems that you can't have a model-selection criterion achieve both consistency (tend to pick the correct model, if there is indeed a correct model and it is among the models being considered) and efficiency (achieve the lowest mean squared error on average among the models you picked). If you tend to pick the right model on average, sometimes you'll get slightly-too-small models... but by often missing a real predictor, you do worse in terms of MSE than someone who always includes a few spurious predictors.

So, as said before, if you care about making-good-predictions more than getting-exactly-the-right-variables, it's fine to keep using LOOCV or AIC.

2) But I also wanted to point out two other of his papers: Yang (2006) "Comparing Learning Methods for Classification" and Yang (2007) "Consistency of Cross Validation for Comparing Regression Procedures". These papers show that you don't need the ratio of training-to-testing data to shrink towards 0 if you're comparing models which converge at slower rates than linear models do.

So, to answer your original questions 1-6 more directly: Shao's results apply when comparing linear models to each other. Whether for regression or classification, if you are comparing nonparametric models that converge at a slower rate (or even comparing one linear model to one nonparametric model), you can use most of the data for training and still have model-selection-consistent CV... but still, Yang suggests that LOOCV is too extreme.

Answer 5

I believe Shao's article applies most effectively to situations where a person is trying to eliminate predictors from a model (whether linear or non-linear, such as machine learning, etc.), and he recommends using Monte Carlo CV (MCCV) in this case.

On the other hand, if you are not worried about the size (number of factors/predictors) in your model, LOOCV might be a shorter and somewhat more intuitive method, which is probably why it is used and accepted so widely.