Implement cross validation for a prediction model

I am trying to assess the predictive performance of two competing linear regression models.

$$model 1: Y \sim X_{1} + X_{2}$$ $$model 2: Y \sim X_{1} + X_{2} + X_{3}$$

where y is continuous.

I would like to estimate the mean squared error (mse) using K-fold cross-validation.

To do this, do I:

1. Get predictions (and mse) on each fold using the coefficient estimates, from mod 1 and 2 from fitting on the full dataset,

or

1. do i calculate a new set of coefficient estimates by re-fitting the model within the fold / CV process and use these to predict on the hold-out set.

or

1. do I use the make predictions on bootstrap resamples of the original data using the models from the full data

Two is the best procedure. To understand this, it helps to consider the sources of randomness that are at play when fitting and evaluating a model, all of which contribute to the expected out of sample error you are attempting to estimate:

• The randomness in $Y$ when $X$ is fixed.
• The randomness in the fit classifier due to fluctuations in the data used to train the classifier.
• The randomness in the error of a fixed classifier, due to the evaluation data differing from the training data.

Procedures one and three ignore the second source of randomness. This is important to account for because although in the end you will have to go forth with a fixed set of coefficient estimates, these fixed estimates are sampled from some distribution around the true estimates (which are unknowable). It is best to make sure your estimation of model performance is aware of this.

I think you have understood my uncertainty; I initially thought of using CV for your reason 2, but as I am actually going to use these estimates, from the full model, to make predictions on another data set (with unknown outcome), I thought perhaps the bootstrap might be the way to go.. I must say, I am still struggling with how CV evaluates my initial model coefficients, rather than the modelling procedure.

I think your understanding is pretty good. Cross validation does evaluate your model procedure, not the final model coefficients, but this is usually the best you can do. Deciding between models 1 and 2 in your setup is a decision about model procedure, or possibly more accurately, model complexity. Cross validation will give you an estimate of the out of sample error for both models, and then you can use it to make an appropriate selection. Once selected, you can re-estiamte the parameters on your full data set, and have confidence that you made the correct choice.

• Thanks for your answer. I think you have understood my uncertainty; I initially thought of using CV for your reason 2, but as I am actually going to use these estimates, from the full model, to make predictions on another data set (with unknown outcome), I thought perhaps the bootstrap might be the way to go.. I must say, I am still struggling with how CV evaluates my initial model coefficients, rather than the modelling procedure. – user2957945 May 23 '15 at 20:25
• I think your understanding is pretty good. Cross validation does evaluate your model procedure, not the final model coefficients, but this is usually the best you can do. Deciding between models 1 and 2 in your setup is a decision about model procedure, or possibly more accurately, model complexity. Cross validation will give you an estimate of the out of sample error for both models, and then you can use it to make an appropriate selection. Once selected, you can re-estiamte the parameters on your full data set, and have confidence that you made the correct choice. – Matthew Drury May 23 '15 at 20:31
• Of course, glad to help. I'll add this little discussion to my answer. – Matthew Drury May 23 '15 at 20:37

In k-fold cross-validation, the original sample is randomly partitioned into $k$ equal sized subsamples. Of the $k$ subsamples, a single subsample is retained as the validation data for testing the model, and the remaining $k − 1$ subsamples are used as training data.

You should train on the rest of the set and predict on the folds. In the end, combine the predictions (obtained across the folds) from each of your models to get the overall prediction $Y_1$ and $Y_2$ from the models $1$ and $2$ respectively. You can then calculate $R^2$ for each of the models and take the decision.