I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I wanted to show my colleagues that there might sometimes be alternatives.
I had a large dataset of approx 30,000 observations. I took 1,000 at random, fitted a model, and then performed the 4 methods of internal validation below to estimate the error. I compared this against the 'true' rate of error from the remaining 29,000 observations. I repeated this whole process 500 times, each time re-sampling 1,000 observations and re-fitting the model etc. The model was an OLS regression with 10 variables.
The results were largely as I had expected: the resubstitution error was optimistically biased, bias and variance in the bootstrap and cross-validation methods was low, and the variance of the single holdout validation method was very high; what I hadn't expected (and what I am at a loss to explain) is the bias I observe in the holdout method. I had assumed it would be high variance, but with low bias. Has anyone else seen this kind of behaviour? Or is it perhaps a consequence of my experimental design?
Clockwise from top left: optimism-corrected bootstrap, resubstitution error, 10-fold CV, single 1:1 holdout
I should note, I'm not concerned here about model selection here - the model had been previously published, and I'm just interested in how it performs when applied to my data.