How to compute (manually) the MSE and $\theta$ with k-fold cross validation in a multiple linear regression

Question

Let's say we have data with predicting variables in a matrix $\textbf{X}$ and a vector of target value $\textbf{Y}$ and we want to find $\pmb{\theta}$ s.t. $$ \arg\min_\theta\frac{1}{n}\sum_{i = 1}^n (X^{(i)} \theta - Y^{(i)})^2 $$ For this we want to use $k$-fold cross-validation to avoid overfitting and have generalizable model. Let's say that $k = 4$. That means for me that we fit 4 different model. The first model fits the subset 1, 2 and 3 and, with the obtained $\theta_1$ compute the Mean Squared Error (MSE) on the subset 4. Then second model fits the subset 1, 2 and 4 and, with the obtained $\theta_2$ compute the Mean Squared Error (MSE) on the subset 3 (and so on for model 3 and 4) let's implement it (in R):

library(ISLR)
library(broom)
library(tidyverse)

rowN = dim(Auto)[1]
subset1 = seq(0, (1*rowN/4), by = 1)
subset2 = seq((1*rowN/4)+1, (2*rowN/4), by = 1)
subset3 = seq((2*rowN/4)+1, (3*rowN/4), by = 1)
subset4 = seq((3*rowN/4)+1, (4*rowN/4), by = 1)


ComputeModel = function(subset1, subset2, subset3){
model=lm(mpg ~ weight + 
            origin + 
            horsepower + 
            year + 
            displacement + 
            acceleration, 
          data=Auto,
          subset=c(subset1, subset2, subset3))
return(model)
}

ComputeTheta = function(model){
  return(tibble(model1$coefficients))
}

ComputeMSE = function(model, subset){
  cat(c("MSE: ", round(mean((Auto$mpg-predict(model,Auto))[subset]^2), 3), "\n"))
}

model1 = ComputeModel(subset1, subset2, subset3)
theta1 = ComputeTheta(model1)
MSE1 = ComputeMSE(model1, subset4)

model2 = ComputeModel(subset1, subset2, subset4)
theta2 = ComputeTheta(model2)
MSE2 = ComputeMSE(model2, subset3)

model3 = ComputeModel(subset1, subset3, subset4)
theta3 = ComputeTheta(model3)
MSE3 = ComputeMSE(model3, subset2)

model4 = ComputeModel(subset2, subset3, subset4)
theta4 = ComputeTheta(model4)
MSE4 = ComputeMSE(model4, subset1)

MSE:  36.138 
MSE:  14.925 
MSE:  10.556 
MSE:  20.411 

Here are my questions: At the end of the $k$-fold cross validation procedure, how do we compute $\pmb{\theta}_{\text{cross-validation}}$ and $\text{MSE}_{\text{cross-validation}}$? Is it simply the mean obtained in the 4 models: $$ \text{MSE}_{\text{cross-validation}} = \frac{\text{MSE}_{\text{model1}}+ \text{MSE}_{\text{model2}}+ \text{MSE}_{\text{model3}}+ \text{MSE}_{\text{model1}}}{4} $$ $$ \pmb{\theta}_{\text{cross-validation}}= \frac{\pmb{\theta}_{\text{model1}}+ \pmb{\theta}_{\text{model2}}+ \pmb{\theta}_{\text{model3}}+ \pmb{\theta}_{\text{model4}} }{4} $$

I read the part regarding this topic in An Introduction to Statistical Learning by James, Hitten, Hastie and Tibshirani, but could not find the details answering the question (or did not understand it properly). Do you have a good read with details on the subject to recommend?

Answer 1

Calculation of MSE cross-validation is typical, but calculating $\theta_{cv}$ as yours is not good in general. Recently, a similar question has been asked; see option (1) in the question. If $X^TX$ is not singular, the problem has only one minimum, and based on your data, you might not see any adverse effects of averaging out $\theta_k$. But still, I wouldn't advise it.

Cross-validation is typically used for two main purposes:

• Tune hyper-parameters (which you don't have)
• Estimate test error (which is MSE-cv you have, and it represents the MSE of future test data, i.e. you claim "we would have trained our model with all the training data, obtain a $\theta$, and calculate MSE on the test set and would have obtained a similar MSE"). This way, you can't claim a final model because you're using your data to report an estimated test error. You can't have your cake and eat it at the same time.