I have this dilemma with CV. I will try to take an example and explain it as best as possible. I'm doing it for lasso regression. I want to find the best coefficient $\beta$ values and the best $\lambda$ value and the prediction error (where I have the dilemma)

  1. I have a dataset 20x10.
  2. To do 5-Fold CV, I split the data into 5 splits: 16 rows of my data are for training and 4 rows for testing.
  3. Over a sequence of e.g. 3 lambda values, $\lambda=1,2,3$.
  4. Starting with $\lambda=1$
  5. First I fit the model on the training data, and get a value of $\beta^{train}$.
  6. I use this $\beta^{train}$ on the test set $X^{test} \beta^{train}-y^{test}$ and I get the MSE, e.g. $MSE_1$.
  7. I repeat this for all 5 folds and $\lambda=1$ and get 5 $MSE$s, so: $MSE_1$,$MSE_2$,$MSE_3$,$MSE_4$,$MSE_5$.
  8. I average them and I get the $CV_1$-error for $\lambda=1$.
  9. I repeat the same process for the remaining $\lambda$ values.
  10. I get for each $\lambda$ value 3 CV-errors: $CV_1$, $CV_2$, $CV_3$.
  11. I check for which $\lambda$ value I get the smallest $CV$, say for $lambda=3$. So I found now the best lambda.

Since the CV error is not the prediction error, how to I measure the prediction error?

Do I have to split the data in 2 parts training and testing and train the model with $\lambda=3$ and I get again a $\beta^{train}$.

After that in the test part I find the $MSE=\frac{1}{n}\sum X^{test} \beta^{train}-y^{test}$ using the $\beta^{train}$ and this is then my prediction error?

Is this correct?

  • 1
    $\begingroup$ If you go with the second approach, you should do that split before your CV. Do the CV on the training set only and then train a model using the best CV parameter but using all the data. This way you know that none of your test set data was used at any stage of the training. However, I think, the CV error is supposed to be an unbiased estimate of the error of training the model on the full set using that parameter. I think. $\endgroup$ – Dan Jun 15 '18 at 15:45
  • $\begingroup$ @Dan should I fist divide the data into training and testing 2 parts. Then do CV on the training data. For CV on the training data, now I divide the training data into sub-training and sub-testing. I do CV and find the optimal $\lambda$. Then I use this $\lambda$ on the test set of the data where I split in the beginning. $\endgroup$ – Ville Jun 15 '18 at 15:50
  • 1
    $\begingroup$ If you think that the CV error is not the prediction error, then yes first split your into a training and test set. Using k-fold cross validation on your training set (i.e. $(k-1)*n_{train}/k$ records in your sub-training set and $n_{train}/k$ records in your sub-testing set where $n_{train}$ is the number of records in your initial training set. Find the CV error for each of your $\lambda$ values by taking the mean over each sub-test fold. Finally, train a model using the $\lambda$ that gave you the lowest CV error on your training set and get the prediction error using your test set. $\endgroup$ – Dan Jun 15 '18 at 15:58
  • 1
    $\begingroup$ Yes, exactly. However even with only 3000 records, your test set is going to be pretty small... $\endgroup$ – Dan Jun 15 '18 at 16:09
  • 1
    $\begingroup$ My point is, that if you think about the standard error of your error metric, it will decrease as your number of samples increases. the CV error is calculated over all 3000 data points but your test set prediction error is calculated over a far smaller sample which is why I say that with a small data-set it might be more correct to use the CV error. Maybe Introduction to Statistical Learning has some insights for you as they cover CV in some detail. $\endgroup$ – Dan Jun 15 '18 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.