1
$\begingroup$

I thought cross validation error would tend to be higher than the test error for the model fit on the entire data set due to the following reasons:

  1. Cross validation uses only a subset of data compared to the model fit on the entire data set, which contributes to bias.

  2. Cross validation averages the outputs of k fitted models in which k is the number of 'folds' in the cross validation, and there's obviously going to be overlaps between training sets, which contribute to the variance.

My guess was that CV error must always be smaller than the test error for the model fit on the entire data set. However, I heard that it is possible for the CV error to be much lower than the test error. Is there some possible reason?

Also, what could be some possible reasons behind CV error to be much lower than the actual theoretical error?

$\endgroup$
2
$\begingroup$

The test error vs CV error depends on the following factors:

  • What is the portion of data used for the CV error? If you would use some 50%, you could get much comparable results as in the test error case.
  • Whether you define the portions of data used for the CV at random or systematically. If you select them at random, it is always quite likely that there will be some data in the neighborhood.
  • Where is the test set. If the test set has input values completely elsewhere than the training set, then (especially in case of models with poor extrapolation such as RBF networks), the error will be much larger than CV error.
| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.