I'm watching a Datacamp video on Machine Learning tool box video with Caret's Zachary Dean Mayer. As he is introducing CV concept, the answer to a multiple choice question on the purpose of CV is (can I post this here?):
Q. What is the advantage of cross-validation over a single train/test split?
A. It gives you multiple estimates of out-of-sample error, rather than a single estimate.
Correct! If all of your estimates give similar outputs, you can be more certain of the model's accuracy. If your estimates give different outputs, that tells you the model does not perform consistently and suggests a problem with it.
The context here is the RMSE of 10 folds which thus will have produced ten models and 10 out of sample estimates of RMSE.
My question. If CV with ten folds on a data set with say 1,000 observations resulted on a predicted value for every observation in the data (at some point during iterating over each fold, each observation is predicted on once, right?) then why would CV not just produce a single RMSE for the dataset as a whole? It's still technically out of sample, isn't it? Because each prediction will have taken place on data not used to build that iteration of the model?
My question short version: Why does CV produce 10 RMSE's and not just 1?