I would like your thoughts about the differences between cross validation and bootstrapping to estimate the prediction error.
Does one work better for small dataset sizes or large datasets?
$\begingroup$
$\endgroup$
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
1
It comes down to variance and bias (as usual). CV tends to be less biased but K-fold CV has fairly large variance. On the other hand, bootstrapping tends to drastically reduce the variance but gives more biased results (they tend to be pessimistic). Other bootstrapping methods have been adapted to deal with the bootstrap bias (such as the 632 and 632+ rules).
Two other approaches would be "Monte Carlo CV" aka "leave-group-out CV" which does many random splits of the data (sort of like mini-training and test splits). Variance is very low for this method and the bias isn't too bad if the percentage of data in the hold-out is low. Also, repeated CV does K-fold several times and averages the results similar to regular K-fold. I'm most partial to this since it keeps the low bias and reduces the variance.
Edit
For large sample sizes, the variance issues become less important and the computational part is more of an issues. I still would stick by repeated CV for small and large sample sizes.
Some relevant research is below (esp Kim and Molinaro).
References
Bengio, Y., & Grandvalet, Y. (2005). Bias in estimating the variance of k-fold cross-validation. Statistical modeling and analysis for complex data problems, 75–95.
Braga-Neto, U. M. (2004). Is cross-validation valid for small-sample microarray classification Bioinformatics, 20(3), 374–380. doi:10.1093/bioinformatics/btg419
Efron, B. (1983). Estimating the error rate of a prediction rule: improvement on cross-validation. Journal of the American Statistical Association, 316–331.
Efron, B., & Tibshirani, R. (1997). Improvements on cross-validation: The. 632+ bootstrap method. Journal of the American Statistical Association, 548–560.
Furlanello, C., Merler, S., Chemini, C., & Rizzoli, A. (1997). An application of the bootstrap 632+ rule to ecological data. WIRN 97.
Jiang, W., & Simon, R. (2007). A comparison of bootstrap methods and an adjusted bootstrap approach for estimating the prediction error in microarray classification. Statistics in Medicine, 26(29), 5320–5334.
Jonathan, P., Krzanowski, W., & McCarthy, W. (2000). On the use of cross-validation to assess performance in multivariate prediction. Statistics and Computing, 10(3), 209–229.
Kim, J.-H. (2009). Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap. Computational Statistics and Data Analysis, 53(11), 3735–3745. doi:10.1016/j.csda.2009.04.009
Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. International Joint Conference on Artificial Intelligence, 14, 1137–1145.
Martin, J., & Hirschberg, D. (1996). Small sample statistics for classification error rates I: Error rate measurements.
Molinaro, A. M. (2005). Prediction error estimation: a comparison of resampling methods. Bioinformatics, 21(15), 3301–3307. doi:10.1093/bioinformatics/bti499
Sauerbrei, W., & Schumacher1, M. (2000). Bootstrap and Cross-Validation to Assess Complexity of Data-Driven Regression Models. Medical Data Analysis, 26–28.
Tibshirani, RJ, & Tibshirani, R. (2009). A bias correction for the minimum error rate in cross-validation. Arxiv preprint arXiv:0908.2904.
-
3$\begingroup$ Bootstrap bias is not pesimistic, it is optimistic (Simple Bootstrap not .0632). This is because Bootstrap uses a lot of training elements to test the model leading to a lot of weight for in sample error. $\endgroup$– D1XCommented Jul 29, 2017 at 12:43
$\begingroup$
$\endgroup$
2
@Frank Harrell has done a lot of work on this question. I don't know of specific references.
But I rather see the two techniques as being for different purposes. Cross validation is a good tool when deciding on the model -- it helps you avoid fooling yourself into thinking that you have a good model when in fact you are overfitting.
When your model is fixed, then using the bootstrap makes more sense (to me at least).
There is an introduction to these concepts (plus permutation tests) using R at http://www.burns-stat.com/pages/Tutor/bootstrap_resampling.html
-
5$\begingroup$ Does it make sense to use CV first to select a model, and after that use bootstrapping on the same data to asses the errors of your estimates? Specifically I want to do linear regression using ML on data with unknown non Gaussian noise. $\endgroup$– sebhoferCommented Oct 10, 2016 at 2:57
-
$\begingroup$ yes, you can select a model with CV and bootstrap is used to define confidence interval of the model. $\endgroup$– JeeyCiCommented Oct 18 at 8:18
$\begingroup$
$\endgroup$
4
My understanding is that bootstrapping is a way to quantify the uncertainty in your model while cross validation is used for model selection and measuring predictive accuracy.
-
$\begingroup$ thanks lot for the answers. I thought bootstrapping was better when you have small data set (<30 obs). No? $\endgroup$– grantCommented Nov 14, 2011 at 17:20
-
$\begingroup$ I would think so. Cross validation may not be reasonable when you have a small sample size. You could do leave one out cross validation but that tends to be overoptimistic. $\endgroup$– GlenCommented Nov 15, 2011 at 14:21
-
$\begingroup$ Also note the doing bootstrapping with a small sample will lead to some biased estimates, as noted in Efron's original paper. $\endgroup$– GlenCommented Nov 18, 2011 at 1:03
-
$\begingroup$ Isn't measuring predictive accuracy a way to quantify uncertainty? I understand CV is more common for model selection, but let's say I want to estimate AUC for a LASSO, is CV or bootstrapping better? $\endgroup$ Commented Mar 4, 2017 at 18:15
$\begingroup$
$\endgroup$
These are two techniques of resampling:
In cross validation we divide the data randomly into kfold and it helps in overfitting, but this approach has its drawback. As it uses random samples so some sample produces major error. In order to minimize CV has techniques but its not so powerful with classification problems. Bootstrap helps in this, it improves the error from its own sample check..for detail please refer..
https://lagunita.stanford.edu/c4x/HumanitiesScience/StatLearning/asset/cv_boot.pdf
