What is cross-validation for?

Question

A very simple question: What is cross-validation for?

As far as I understand, cross-validation is used for selecting the model and not the parameters of the model, but I want to check if I am right. The famous k-fold, illustrated in this image:

uses $k$ combinations of train-test samples to train and test the model and is used to avoid overfitting.

So if we have a model M, Is the model trained from scratch for every train-test combination?

So are these steps correct?

repeat k times:
    train M with sample train[i]
    predict test[i] with M
    compute MeanSquaredError[i] for test[i]
    i = i+1
end repeat
compute mean of MeanmumSquaredError

As the model is re-trained everytime this is only usefull to check if the model is well chosen and not the parameters of the model, right?

---

UPDATE: Let's suppose the model M is a neural network with one hidden layer. Do you use cross-validation for example to select the number of neurons in the hidden layer?

Answer 1

Cross-validation is, in my opinion, a method to estimate performance of your model AND its parameters. It is also a good measure of how robust your model with its parameters is.

Let's say you decided two methods are appropriate for your data: ordinary least squares(OLS) and ridge regression.

For ridge regression case, there is a parameter called lambda used in regularization in terms of sum of coefficents. How can you decide what value of lambda provides the best model? This is where the CV comes in. You can now apply cross-validation to calculate the MSE with different lambdas and select the lambda value where increasing it further doesn't improve your model. Thus, CV can be used for parameter optimization.

The other use is to compare models i.e. OLS and ridge, so after optimizing parameters, you can compare the models by their CV errors. This is, however, quite risky and I wouldn't recommend it because even though CV provides some insight about the model's success, there is no way to be sure how exactly good your model is. An example of possible risk is selecting a model parameter that overfits to your training set and fail on new data.

I didn't full understand your pseudo code so here is an example for leave-one-out-cross-validation(k-fold CV where k = number of samples):

for i=1:number of samples
   leave ith sample out so that you now have n-1 samples
   build model with remaining samples
   predict the ith sample (left out sample)
   calculate its error
   put the ith sample back in to have the original matrices again
end
calculate total error (MSE,RMSE or whichever measure you think is appropriate)

Edit: The answer to the comment is no. CV models shouldn't be used. Instead, one should build the final model with whole training set.

Answer 2

Cross-validation is, obviously, for validation.

It results with some measure of how good your model is (by the way, I'd use MeanSquaredError instead of MinimumSquaredError, whatever it is). So it enables you to assess model's future performance and to compare models.

Of, course, you can assess MSE without cross-validation, but this would be biased, since you'll have to use the same sample to train and to validate your model.

As the model is re-trained everytime this is only usefull to check if the model is well chosen and not the parameters of the model, right?

Right. Almost.

Imagine, your train procedure contains some feature selection method. Then you can not compare model with pre-selected features via cross-validation (e.g. you can not choose between Y~X+Z and Y~X+V), beacuse your train method will select different features for different fold.

On the other hand, imagine your train procedure estimates all the parameters instead of one (say, $\beta_0$). Then you can compare models with $\beta_0=1$ and with $\beta_0=2$ via cross-validation.

So, to sum up: you can compare models that differ in parameters that are not estimated inside train method (have to be inputed to it).

Answer 3

Cross-validation is a method to validate a model, which is used mostly in cases when you have a very limited amount of data available. You never want to train on data on which you are validating. On the other hand, sometimes it is costly to totally remove part of the training set (for validation). Cross-validation is a middle-ground here. We use some part of training set for validation, but never train and validate on the same data in the same time. Cross-validation is an elegant solution to achieve that.