3
$\begingroup$

In the documentation for the R package pls, the following statement appears in the help file for the MSEP function:

"CV" is the cross-validation estimate, and "adjCV" (for RMSEP and MSEP) is the bias-corrected cross-validation estimate.

What does this mean? In other words, what does it mean to correct a CV estimate for bias?

$\endgroup$
3
$\begingroup$

In this context, bias correction refers to the fact that, when we do perform resampling (bootstrap or cross-validation) we almost certainly do not use our whole sample of size $N$; this potential leads to the biased estimates of the MSEP (Mean Squared Error of Prediction).

There are various methodologies that can control for this kind of resampling bias. For example, one of the mostly commonly referenced techniques is the bootstrap 0.632 (Efron, 1983, JASA, Sect. 6). What all methodologies have in common is that they derive a relation approximating the expected difference in performance between a learner trained with the "resampled sample" and another ideal learner trained with the full sample. They then recombine/weight the estimates in such way that the apparent discrepancy is minimised. For example, the adjCV estimator, as implemented in pls::MSEP, adjusts by a factor proportional to the difference of the whole sample MSEP and mean out-of-fold MSEP (see Mevik & Cederkvist, 2005, Chemometrics, Sect. 2.4 ). Similarly, the bootstrap 0.632 estimator recombines the out-of-bootstrap-sample error estimate with the in-bootstrap-sample error estimate.

A nice succinct introduction to the topic touching among the issue of bias (and variance) can be found in Sections 7.10 (Cross-validation) and 7.11 (Bootstrap Methods) from Hastie et al.'s classic textbook Elements of Statistical Learning, they touch upon bias mostly in the context of bootstrap 0.632 but the rationale for biased adjusted CV is the same. Finally, the CV community has already two very enlightening posts regarding: What is the .632+ rule in bootstrapping? and Bias and variance in leave-one-out vs K-fold cross validation; they definitely worth one's time! (Personal-note: People tend to make a big issue about bias but I have found that the variance is the one that often kills an analysis.)

$\endgroup$
  • $\begingroup$ Makes perfect sense. Great answer! $\endgroup$ – Sean Raleigh Oct 11 '18 at 3:05
  • $\begingroup$ Cool, I am happy I could help. $\endgroup$ – usεr11852 Oct 11 '18 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.