1
$\begingroup$

I'm watching a Datacamp video on Machine Learning tool box video with Caret's Zachary Dean Mayer. As he is introducing CV concept, the answer to a multiple choice question on the purpose of CV is (can I post this here?):

Q. What is the advantage of cross-validation over a single train/test split?

A. It gives you multiple estimates of out-of-sample error, rather than a single estimate.

Correct! If all of your estimates give similar outputs, you can be more certain of the model's accuracy. If your estimates give different outputs, that tells you the model does not perform consistently and suggests a problem with it.

The context here is the RMSE of 10 folds which thus will have produced ten models and 10 out of sample estimates of RMSE.

My question. If CV with ten folds on a data set with say 1,000 observations resulted on a predicted value for every observation in the data (at some point during iterating over each fold, each observation is predicted on once, right?) then why would CV not just produce a single RMSE for the dataset as a whole? It's still technically out of sample, isn't it? Because each prediction will have taken place on data not used to build that iteration of the model?

My question short version: Why does CV produce 10 RMSE's and not just 1?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

"Pooling" all of the predictions and calculating the error over the entrie dataset as opposed to calculating the error for each fold and averaging results in a biased estimate.

The bias may be minimal most of the time and for metrics like RMSE but it can be quite significant for rank based performance measures as discussed here: https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-8-326

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you. Now I know. But is it biased even if each prediction was calculated out of sample on a given fold? I.e. each prediction was made on data that was not used in the model on a given iteration. $\endgroup$
    – Doug Fir
    Commented Apr 12, 2017 at 3:46
  • 2
    $\begingroup$ The folds aren't independent samples. Thinking of repeated two fold validation, the mean of the target variable will be anti correlated between the two folds (say it's determined by which fold the high value cases ended up in) which will particularly bias ranking and rank based measures. But awareness of this bias is actually somewhat uncommon. People do pool their predictions and calculate error over that quite and the bias is often not an issue especially with non rank based measures like RMSE. $\endgroup$
    – Ryan Bressler
    Commented Apr 12, 2017 at 4:44
  • 2
    $\begingroup$ Thanks for the additional explanation. I think I follow yet remain a bit confused. I understand what you mean about some potentially distorting samples ending up in one fold, but I cannot get my head around how each prediction is no more or less biased than an non cv simple 80/20 split. Anyway, thanks for your answer $\endgroup$
    – Doug Fir
    Commented Apr 12, 2017 at 4:53
  • $\begingroup$ @RyanBressler What do you mean with bias? Bias with respect to what? In fact $(\sqrt{X} + \sqrt{Y})/2 \leq \sqrt{(X + Y)/2}$ specifically for RootMSE. $\endgroup$
    – Jim
    Commented May 23, 2017 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.