I have a bit of a misunderstanding of what sample is being used to calculate the MSE each time in the procedure for LOOCV. I believe that it is the training set rather than the test set. Is the training set or test set being used to calculate the MSE in the procedure for LOOCV?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Why do you think that it's calculated from the training set? $\endgroup$– hbadger19042Commented Jul 20, 2020 at 14:53
-
$\begingroup$ Now I'm thinking it might be the test set because this is one that we are testing our predictions. Thus, we would want to determine accuracy from this validation set/training set. Maybe this isn't the correct reasoning. $\endgroup$– KliocontarCommented Jul 20, 2020 at 15:10
-
$\begingroup$ CV is not done on a holdout test set. $\endgroup$– gunesCommented Jul 20, 2020 at 15:22
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
In Leave-one-out cross validation (LOOCV) method, for each observation in our sample, say the $i$-th one, we first fit the same model keeping aside the $i$-th observation and then calculate the mean squared error for the $i$-th observation. Finally we take the average of these individual mean squared errors.
For example, suppose our model is $Y = f(X) + \varepsilon$ and we have some estimate for $f,$ say $\hat{f},$ which is computed on the basis of all observations. Now in LOOCV method, we calculate $\hat{f}$ after deleting the $i$-th observation from our dataset, let's call it $\hat{f}_{-i}(x)$ and then compute $(y_i - \hat{f}_{-i}(x_i))^2.$ Finally we compute the average of these quantities.
-
$\begingroup$ Ah ok, so it is the test set that we're really calculating the MSE on for LOOCV. Right? This would be the ith observation basically. $\endgroup$ Commented Jul 20, 2020 at 15:16
-
$\begingroup$ I just added an example, see above. $\endgroup$ Commented Jul 20, 2020 at 15:17
-
$\begingroup$ Yes we calculate the MSE on the test set. But the key idea in cross validation is to divide the whole sample into train data and test data and doing it for every possible manner we divide the sample. (I mean, we don't have any extra test data, we pick the test data from the sample itself.) $\endgroup$ Commented Jul 20, 2020 at 15:19
-
$\begingroup$
$\endgroup$
I gather it is the same principle found in k-fold. LOOCV is (sort of) specific case. Out of the dataset, a parcel is used as training and one point is left out, sucessively. Then, the average is calculated. In short, the same data set is used as train and test, on account of successive modifications of its partition.
