2
$\begingroup$

Let's say we have data with predicting variables in a matrix $\textbf{X}$ and a vector of target value $\textbf{Y}$ and we want to find $\pmb{\theta}$ s.t. $$ \arg\min_\theta\frac{1}{n}\sum_{i = 1}^n (X^{(i)} \theta - Y^{(i)})^2 $$ For this we want to use $k$-fold cross-validation to avoid overfitting and have generalizable model. Let's say that $k = 4$. That means for me that we fit 4 different model. The first model fits the subset 1, 2 and 3 and, with the obtained $\theta_1$ compute the Mean Squared Error (MSE) on the subset 4. Then second model fits the subset 1, 2 and 4 and, with the obtained $\theta_2$ compute the Mean Squared Error (MSE) on the subset 3 (and so on for model 3 and 4) let's implement it (in R):

library(ISLR)
library(broom)
library(tidyverse)

rowN = dim(Auto)[1]
subset1 = seq(0, (1*rowN/4), by = 1)
subset2 = seq((1*rowN/4)+1, (2*rowN/4), by = 1)
subset3 = seq((2*rowN/4)+1, (3*rowN/4), by = 1)
subset4 = seq((3*rowN/4)+1, (4*rowN/4), by = 1)


ComputeModel = function(subset1, subset2, subset3){
model=lm(mpg ~ weight + 
            origin + 
            horsepower + 
            year + 
            displacement + 
            acceleration, 
          data=Auto,
          subset=c(subset1, subset2, subset3))
return(model)
}

ComputeTheta = function(model){
  return(tibble(model1$coefficients))
}

ComputeMSE = function(model, subset){
  cat(c("MSE: ", round(mean((Auto$mpg-predict(model,Auto))[subset]^2), 3), "\n"))
}

model1 = ComputeModel(subset1, subset2, subset3)
theta1 = ComputeTheta(model1)
MSE1 = ComputeMSE(model1, subset4)

model2 = ComputeModel(subset1, subset2, subset4)
theta2 = ComputeTheta(model2)
MSE2 = ComputeMSE(model2, subset3)

model3 = ComputeModel(subset1, subset3, subset4)
theta3 = ComputeTheta(model3)
MSE3 = ComputeMSE(model3, subset2)

model4 = ComputeModel(subset2, subset3, subset4)
theta4 = ComputeTheta(model4)
MSE4 = ComputeMSE(model4, subset1)

MSE:  36.138 
MSE:  14.925 
MSE:  10.556 
MSE:  20.411 

Here are my questions: At the end of the $k$-fold cross validation procedure, how do we compute $\pmb{\theta}_{\text{cross-validation}}$ and $\text{MSE}_{\text{cross-validation}}$? Is it simply the mean obtained in the 4 models: $$ \text{MSE}_{\text{cross-validation}} = \frac{\text{MSE}_{\text{model1}}+ \text{MSE}_{\text{model2}}+ \text{MSE}_{\text{model3}}+ \text{MSE}_{\text{model1}}}{4} $$ $$ \pmb{\theta}_{\text{cross-validation}}= \frac{\pmb{\theta}_{\text{model1}}+ \pmb{\theta}_{\text{model2}}+ \pmb{\theta}_{\text{model3}}+ \pmb{\theta}_{\text{model4}} }{4} $$

I read the part regarding this topic in An Introduction to Statistical Learning by James, Hitten, Hastie and Tibshirani, but could not find the details answering the question (or did not understand it properly). Do you have a good read with details on the subject to recommend?

$\endgroup$
1
$\begingroup$

Calculation of MSE cross-validation is typical, but calculating $\theta_{cv}$ as yours is not good in general. Recently, a similar question has been asked; see option (1) in the question. If $X^TX$ is not singular, the problem has only one minimum, and based on your data, you might not see any adverse effects of averaging out $\theta_k$. But still, I wouldn't advise it.

Cross-validation is typically used for two main purposes:

  • Tune hyper-parameters (which you don't have)
  • Estimate test error (which is MSE-cv you have, and it represents the MSE of future test data, i.e. you claim "we would have trained our model with all the training data, obtain a $\theta$, and calculate MSE on the test set and would have obtained a similar MSE"). This way, you can't claim a final model because you're using your data to report an estimated test error. You can't have your cake and eat it at the same time.
$\endgroup$
7
  • $\begingroup$ Thank you for your answer and the link @gunes. As I added in the question I read the part regarding this topic in "An Introduction to Statistical Learning by James, Hitten, Hastie and Tibshirani (faculty.marshall.usc.edu/gareth-james/ISL), but could not find the details answering the question (or did not understand it properly). Do you have a good read with details on the subject to recommend? $\endgroup$
    – ecjb
    Jun 17 '20 at 6:57
  • $\begingroup$ You may not see it explicitly in the books because for the parameters you’re doing an unusual thing. But, you should be seeing MSE cv in various books/posts/tutorials. $\endgroup$
    – gunes
    Jun 17 '20 at 11:00
  • $\begingroup$ @ecjb is the answer ok for you or is there anything that is unclear? $\endgroup$
    – gunes
    Jun 23 '20 at 15:48
  • $\begingroup$ Yes your linked answer gave me some hints. What really helped me understand the big picture of the problem though (I think) was the p. 242 of "The Elements of Statistical Learning" (web.stanford.edu/~hastie/Papers/ESLII.pdf). As I understood it, you would fit a model and test it as follows: first you perform k-fold cross validation on a training set to select the optimal $\lambda$ (with the smallest error). Then you fit a model on the whole training set using the obtained optimal lambda. And then on the test set you compute the error of the model. Correct? $\endgroup$
    – ecjb
    Jun 23 '20 at 19:03
  • $\begingroup$ That is true, and follows from the first option I've listed above. But to add, if you don't want to test it using a separate data (when your dataset is small), you do CV and estimate the test error as you've done. This way, you can't use the same folds for HP tuning. You need nested folds. $\endgroup$
    – gunes
    Jun 23 '20 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.