What is the uncertainty of Leave-one-out-cross-validation method?

Question

I have used the LOOCV to validate my model. As we know, the LOOCV method is a special case of cross-validation where the number of folds equals the number of instances in the data set. Thus, the learning algorithm is applied once for each instance, using all other instances as a training set and using the selected instance as a single-item test set.

My datasets including 50 samples. After done the LOOCV, I get 50 pairs of observed data and corresponding predicted data. Then, I did the accuracy assessment by compute the R2 and RMSE of these 50 pairs data. But someone suggested me to comment on the uncertainty in the RMSE and R2 measurements. Actually, I didn't understand what is the uncertainty mean.

Are there anyone could help me? Thanks so much!

I computed the R2 and RMSE by

where E_i and O_i denote the estimated and observed value, respectively, E ̅ and O ̅ denote the mean of the estimated and observed value, respectively, and n is the sample size (i.e., 50).

Answer 1

Actually, I didn't understand what is the uncertainty mean.

Maybe think like this: You try to measure predictive performance, and like any other measurement result, also your results are subject to random and systematic uncertainty.

Imagine for a moment that you had only 5 such predicted/reference pairs, or 500. Clearly, the RMSE based on only 5 pairs (= tested cases) is less well known, i.e., more uncertain, than an RMSE based on 50 or 500 pairs.

Now, RMSE contains several "sources" of prediction error: bias, i.e. the predictions being systematically off (which does not enter R²), and several sources of random error, i.e. variance.
Two of the latter include the variance uncertainty caused by testing only a limited number of cases (50) - that's the variance I explained above.
In addition, there is potential variation between your surrogate models: they may not yield totally stable predictions. I.e., two such models may yield different predictions for the same cases (imagine you acquire an additional case and let it be predicted by all 50 models).

Unfortunately, leave-one-out cross-validation leaves you with surrogate model and test case being collinear, so you cannot distinguish between the two. k-fold cross-validation, and even better repeated k-fold CV, would allow to separate these two sources of variance.

In addition, how and which important sources of variance enter the uncertainty of your RMSE estimate depends on your precise application scenario. In particular, on whether you use the cross validation as approximation to measure the RMSE of the training algorithm for data such as yours, or whether you approximate the RMSE of the actual model you obtain on all data points for production use. As you see, a quantitative discussion of these sources of uncertainty on your model performance estimate is quite difficult.
There is also some bias expected in the RMSE estimation via cross validation. If all is well (in particular with your splitting procedure), this should be small and pessimistic.

What you can and should do, though, is to discuss which of these sources are relevant in your case.

---

You can also have a jab at the sample/surrogate model instability based on jack-knifing (i.e. in turn leaving one pair out of the RMSE calculation) or bootstrapping your RMSE (or R²) calculations. However, this will typically not include all relevant sources of uncertainty on the model performance estimate. It does give you a low guesstimate, though.