Meaning of cross validation

Question

This is a very fundamental question but I want to make sure I get this right.

K-fold cross validation will only help in predicting the accuracy and other metrics of the model but not really improve the model. Is that correct?

I am trying to read and learn about it and what I find is this is the approach mainly to have a better estimate of the model performance than applying it on single data set. However, CV cannot directly help in model improvement as in it will not give out a better model. You would have to tune the model again through different algorithms and again run it through CV to compare performance.

Is my understanding correct?

Answer 1

You are right (for K-fold CV)

K-fold CV is a metric that would provide a better measure of accuracy for your model than a regular single training-testing split. It would not do more than that. It would have no effect on the learning itself.

However, cross validation or (just validation) is used in conjunction with learning in different learning algorithms in different ways that will indeed aid in better learning by preventing overfitting. This is a separate set of called samples called the validation set disjoint from the training and testing sets for this purpose.

For instance, in ANN, the the accuracy of validation set is measured at every epoch to halt the training at the point of overfitting. This procedure is called early stopping.

Similarly validation sets are used to prune decision trees after allowing them to overfit.

Squares plot the accuracy of the training set while triangles indicate accuracy of the validation set.

Don't get confused between the two terms as they are used in different context.

Answer 2

Yes, you are correct. Cross validation will give you an idea of your model's out-of-sample performance. It does not modify the model as such.

Typically, you will look at multiple models when you want to predict. (The situation is somewhat different when doing inferential statistics.) You will cross validate each model and pick the one that yields the lower cross validated error. In this way, you can examine the effects of adding, removing or transforming predictors.

Cross validation is a tool to help you understand your model better. You will need to do the modifying of your model yourself - guided by what cross validation tells you.