clarification on k fold cross validation

Question

I understand k fold cross validation has been discussed many times here and I read some of them. I understand how k Fold works, but what confuses me is the following:

Cross validation in my understanding is used for model validation, which validates an existing model. For example, a model builds on full dataset. But what I read about k fold is that it will build 5 different models based on 4/5 of the full date set (sampled 5 times) and test it on the rest 1/5 dataset, if k =5. It seems it has nothing to do with validating an existing model.

Could anyone kindly comment on this?

Answer 1

Cross-validation estimates the performance of a method of producing a model, not of the model itself. The performance of the model can reasonably be expected to be consistent with the performance of models generated by this method, but the indirect inference there is usually not stated.

Answer 2

I like this question because it touches a common source of confusion: the overloaded use of the term model. In this contest, model refers to a the mathematical formulation of a solution (a method $ \hat f (x, \theta)$) rather than a specific instance of a model where every parameter ($\theta$) has been set.

I have also seen the k trained model instances generated in a k-fold cross validation exercise moved to production and used as an ensemble instead of training one, single model on the whole calibration set.

This is a particularly concerning topic, because as the number of folds declines, depending on the performance of the learning method with respect to the training set size, might result on k-fold model instances being biased - not only in their estimate of the generalization error but, in this specific context, biased in their estimation of the target variable.

---

Edit 1

If I understand what you said correctly, k fold validation is more used to compare modeling algorithms, but a specific model itself. What I am not quite understand is the bias you talked about. Would you please comment on that? Thanks!

Your understanding is correct and the comparison takes place on the generalization error, which we estimate calculating the error, or cost, on each validation set and then (very likely) averaging it.

As every estimator, our estimator of the generalization error suffers of bias and variance, which we have to trade-off. An Introduction to Statistical Learning covers this topic in detail. Some other treads explain this thoroughly: this and this for example.