In his paper Linear Model Selection by Cross-Validation, Jun Shao shows that for the problem of variable selection in multivariate linear regression, the method of leave-one-out cross validation (LOOCV) is 'asymptotically inconsistent'. In plain English, it tends to select models with too many variables. In a simulation study, Shao shows that even for as few as 40 observations, LOOCV can underperform other cross-validation techniques.
This paper is somewhat controversial, and somewhat ignored (10 years after its publication, my chemometrics colleagues had never heard of it and were happily using LOOCV for variable selection...). There is also a belief (I am guilty of this), that its results extend somewhat beyond the original limited scope.
The question, then: how far do these results extend? Are they applicable to the following problems?
- Variable selection for logistic regression/GLM?
- Variable selection for Fisher LDA classification?
- Variable selection using SVM with finite (or infinite) kernel space?
- Comparison of models in classification, say SVM using different kernels?
- Comparison of models in linear regression, say comparing MLR to Ridge Regression?
- etc.