97
$\begingroup$

I would like your thoughts about the differences between cross validation and bootstrapping to estimate the prediction error.

Does one work better for small dataset sizes or large datasets?

$\endgroup$
84
$\begingroup$

It comes down to variance and bias (as usual). CV tends to be less biased but K-fold CV has fairly large variance. On the other hand, bootstrapping tends to drastically reduce the variance but gives more biased results (they tend to be pessimistic). Other bootstrapping methods have been adapted to deal with the bootstrap bias (such as the 632 and 632+ rules).

Two other approaches would be "Monte Carlo CV" aka "leave-group-out CV" which does many random splits of the data (sort of like mini-training and test splits). Variance is very low for this method and the bias isn't too bad if the percentage of data in the hold-out is low. Also, repeated CV does K-fold several times and averages the results similar to regular K-fold. I'm most partial to this since it keeps the low bias and reduces the variance.

Edit

For large sample sizes, the variance issues become less important and the computational part is more of an issues. I still would stick by repeated CV for small and large sample sizes.

Some relevant research is below (esp Kim and Molinaro).

References

Bengio, Y., & Grandvalet, Y. (2005). Bias in estimating the variance of k-fold cross-validation. Statistical modeling and analysis for complex data problems, 75–95.

Braga-Neto, U. M. (2004). Is cross-validation valid for small-sample microarray classification Bioinformatics, 20(3), 374–380. doi:10.1093/bioinformatics/btg419

Efron, B. (1983). Estimating the error rate of a prediction rule: improvement on cross-validation. Journal of the American Statistical Association, 316–331.

Efron, B., & Tibshirani, R. (1997). Improvements on cross-validation: The. 632+ bootstrap method. Journal of the American Statistical Association, 548–560.

Furlanello, C., Merler, S., Chemini, C., & Rizzoli, A. (1997). An application of the bootstrap 632+ rule to ecological data. WIRN 97.

Jiang, W., & Simon, R. (2007). A comparison of bootstrap methods and an adjusted bootstrap approach for estimating the prediction error in microarray classification. Statistics in Medicine, 26(29), 5320–5334.

Jonathan, P., Krzanowski, W., & McCarthy, W. (2000). On the use of cross-validation to assess performance in multivariate prediction. Statistics and Computing, 10(3), 209–229.

Kim, J.-H. (2009). Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap. Computational Statistics and Data Analysis, 53(11), 3735–3745. doi:10.1016/j.csda.2009.04.009

Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. International Joint Conference on Artificial Intelligence, 14, 1137–1145.

Martin, J., & Hirschberg, D. (1996). Small sample statistics for classification error rates I: Error rate measurements.

Molinaro, A. M. (2005). Prediction error estimation: a comparison of resampling methods. Bioinformatics, 21(15), 3301–3307. doi:10.1093/bioinformatics/bti499

Sauerbrei, W., & Schumacher1, M. (2000). Bootstrap and Cross-Validation to Assess Complexity of Data-Driven Regression Models. Medical Data Analysis, 26–28.

Tibshirani, RJ, & Tibshirani, R. (2009). A bias correction for the minimum error rate in cross-validation. Arxiv preprint arXiv:0908.2904.

$\endgroup$
  • 1
    $\begingroup$ Bootstrap bias is not pesimistic, it is optimistic (Simple Bootstrap not .0632). This is because Bootstrap uses a lot of training elements to test the model leading to a lot of weight for in sample error. $\endgroup$ – D1X Jul 29 '17 at 12:43
33
$\begingroup$

@Frank Harrell has done a lot of work on this question. I don't know of specific references.

But I rather see the two techniques as being for different purposes. Cross validation is a good tool when deciding on the model -- it helps you avoid fooling yourself into thinking that you have a good model when in fact you are overfitting.

When your model is fixed, then using the bootstrap makes more sense (to me at least).

There is an introduction to these concepts (plus permutation tests) using R at http://www.burns-stat.com/pages/Tutor/bootstrap_resampling.html

$\endgroup$
  • 1
    $\begingroup$ Does it make sense to use CV first to select a model, and after that use bootstrapping on the same data to asses the errors of your estimates? Specifically I want to do linear regression using ML on data with unknown non Gaussian noise. $\endgroup$ – sebhofer Oct 10 '16 at 2:57
9
$\begingroup$

My understanding is that bootstrapping is a way to quantify the uncertainty in your model while cross validation is used for model selection and measuring predictive accuracy.

$\endgroup$
  • $\begingroup$ thanks lot for the answers. I thought bootstrapping was better when you have small data set (<30 obs). No? $\endgroup$ – grant Nov 14 '11 at 17:20
  • $\begingroup$ I would think so. Cross validation may not be reasonable when you have a small sample size. You could do leave one out cross validation but that tends to be overoptimistic. $\endgroup$ – Glen Nov 15 '11 at 14:21
  • $\begingroup$ Also note the doing bootstrapping with a small sample will lead to some biased estimates, as noted in Efron's original paper. $\endgroup$ – Glen Nov 18 '11 at 1:03
  • $\begingroup$ Isn't measuring predictive accuracy a way to quantify uncertainty? I understand CV is more common for model selection, but let's say I want to estimate AUC for a LASSO, is CV or bootstrapping better? $\endgroup$ – Max Ghenis Mar 4 '17 at 18:15
9
$\begingroup$

One difference is that cross-validation, like jackknife, uses all of your data points, whereas bootstrapping, which resamples your data randomly, may not hit all the points.

You can bootstrap as long as you want, meaning a larger resample, which should help with smaller samples.

The cross-validation or jackknife mean will be the same as the sample mean, whereas the bootstrap mean is very unlikely to be the same as the sample mean.

As cross-validation and jackknife weight all the sample points the same, they should have a smaller (though possibly incorrect) confidence interval than bootstrap.

$\endgroup$
  • 2
    $\begingroup$ Neil, seems at least 2 of 4 your statements are wrong. 1. Even though each particular boostrap sample covers ~63% of original datapoints, if we sample many (e.g. 10k) bootstrap samples as we usually do, the chance that each point will be covered in at least one of them is essentially 100%. 2. I just did a quick numerical check - the average of bootstrap and out-of-bootstrap samples is very close to the whole data average. You can check yourself $\endgroup$ – Kochede Dec 27 '16 at 5:37
  • $\begingroup$ Here's a code (click "Edit" to see it formatted): import numpy as np, pandas as pd n=1000 B=1000 y = np.random.randn( n ) meansb, meansoob = [], [] for b in range(B): ib = np.random.choice( n, n, replace=True ) meanb = y[ ib ].mean() meansb.append( meanb ) indoob = np.ones(n, dtype=bool) indoob[ ib ] = False meanoob = y[ indoob ].mean() meansoob.append( meanoob ) pd.Series( meansb ).hist( histtype='step' ) pd.Series( meansoob ).hist( histtype='step' ) print np.mean( meansb ), np.mean( meansoob ), pd.Series( y ).mean() $\endgroup$ – Kochede Dec 27 '16 at 5:37
  • $\begingroup$ @Kochede "essentially 100%" is not 100%. "Very close to the average" is not the same as exactly the same as the average. You are using weasel words. I am not wrong $\endgroup$ – Neil McGuigan Nov 2 '17 at 21:02
  • $\begingroup$ So you not only give wrong answers, but you also insist on them with demagogy, ok. Expected coverage of data by bootstrap samples quickly converge to 100% with increasing number of samples. In the above experiment after less than 10 bootstrap samples all datapoints are hit by bootstrap. Similarly, expectation of average of bootstrap samples is equal to sample mean. And in the above experiment after 1000 bootstrap samples difference is less than 0.1% (use np.random.rand instead of randn, because for randn mean is 0) $\endgroup$ – Kochede Dec 8 '17 at 2:14
  • $\begingroup$ Here is the updated code for your reference: import numpy as np, pandas as pd; n=1000; B=1000; y = np.random.rand( n ); meansb = []; covered = np.zeros(n, dtype=bool); coverage = []; #begin loop for b in range(B): ib = np.random.choice( n, n, replace=True ); covered[ ib ] = True; coverage.append( covered.sum() * 1.0 / n ); meanb = y[ ib ].mean(); meansb.append( meanb ); #end loop print coverage[:10]; print np.mean( meansb ), pd.Series( y ).mean(); print (np.mean( meansb ) - pd.Series( y ).mean() ) / pd.Series( y ).mean(); $\endgroup$ – Kochede Dec 8 '17 at 2:49
2
$\begingroup$

These are two techniques of resampling:

In cross validation we divide the data randomly into kfold and it helps in overfitting, but this approach has its drawback. As it uses random samples so some sample produces major error. In order to minimize CV has techniques but its not so powerful with classification problems. Bootstrap helps in this, it improves the error from its own sample check..for detail please refer..

https://lagunita.stanford.edu/c4x/HumanitiesScience/StatLearning/asset/cv_boot.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.