0
$\begingroup$

I have used the LOOCV to validate my model. As we know, the LOOCV method is a special case of cross-validation where the number of folds equals the number of instances in the data set. Thus, the learning algorithm is applied once for each instance, using all other instances as a training set and using the selected instance as a single-item test set.

My datasets including 50 samples. After done the LOOCV, I get 50 pairs of observed data and corresponding predicted data. Then, I did the accuracy assessment by compute the R2 and RMSE of these 50 pairs data. But someone suggested me to comment on the uncertainty in the RMSE and R2 measurements. Actually, I didn't understand what is the uncertainty mean.

Are there anyone could help me? Thanks so much!

I computed the R2 and RMSE by enter image description here

where E_i and O_i denote the estimated and observed value, respectively, E ̅ and O ̅ denote the mean of the estimated and observed value, respectively, and n is the sample size (i.e., 50).

$\endgroup$
2
  • $\begingroup$ You have to be careful interpreting $R^2$ out of sample. It tells you about the spread of your prediction errors but ignores any bias. $\endgroup$ Commented May 26, 2022 at 14:06
  • $\begingroup$ That depends on how you calculate out-of-sample $R^2$. @RichardHardy is spot-on in saying that $cor(y_{test}, \hat y_{test})$ ignores systemic bias (consistently predicting too high or too low). This is why I would go with a calculation based on the idea of comparing the model SSE to the SSE of a model that always predicts the in-sample mean. This avoids the bias issue. The downside is that it is not as typical in software, and it might be harder to explain to stakeholders (clients, customers, bosses, etc). $\endgroup$
    – Dave
    Commented May 26, 2022 at 14:13

1 Answer 1

1
$\begingroup$

Actually, I didn't understand what is the uncertainty mean.

Maybe think like this: You try to measure predictive performance, and like any other measurement result, also your results are subject to random and systematic uncertainty.

Imagine for a moment that you had only 5 such predicted/reference pairs, or 500. Clearly, the RMSE based on only 5 pairs (= tested cases) is less well known, i.e., more uncertain, than an RMSE based on 50 or 500 pairs.

Now, RMSE contains several "sources" of prediction error: bias, i.e. the predictions being systematically off (which does not enter R²), and several sources of random error, i.e. variance.
Two of the latter include the variance uncertainty caused by testing only a limited number of cases (50) - that's the variance I explained above.
In addition, there is potential variation between your surrogate models: they may not yield totally stable predictions. I.e., two such models may yield different predictions for the same cases (imagine you acquire an additional case and let it be predicted by all 50 models).

Unfortunately, leave-one-out cross-validation leaves you with surrogate model and test case being collinear, so you cannot distinguish between the two. k-fold cross-validation, and even better repeated k-fold CV, would allow to separate these two sources of variance.

In addition, how and which important sources of variance enter the uncertainty of your RMSE estimate depends on your precise application scenario. In particular, on whether you use the cross validation as approximation to measure the RMSE of the training algorithm for data such as yours, or whether you approximate the RMSE of the actual model you obtain on all data points for production use. As you see, a quantitative discussion of these sources of uncertainty on your model performance estimate is quite difficult.
There is also some bias expected in the RMSE estimation via cross validation. If all is well (in particular with your splitting procedure), this should be small and pessimistic.

What you can and should do, though, is to discuss which of these sources are relevant in your case.


You can also have a jab at the sample/surrogate model instability based on jack-knifing (i.e. in turn leaving one pair out of the RMSE calculation) or bootstrapping your RMSE (or R²) calculations. However, this will typically not include all relevant sources of uncertainty on the model performance estimate. It does give you a low guesstimate, though.

$\endgroup$
8
  • $\begingroup$ My model is certain for each prediction. For example, when leave the sample 1 as the test data and using samples 2-50 as the train data, in this case, the predicted value is the same whatever how many times I repeat this experiment. Because the train data and the test data are same. $\endgroup$
    – Yanxi Li
    Commented May 27, 2022 at 1:17
  • $\begingroup$ They said that I should provide the standard deviation of RMSE and R2 because the leave-one-out validation method was used. They wonder 'what is the deviation one can expect wrt the given values?'. $\endgroup$
    – Yanxi Li
    Commented May 27, 2022 at 1:19
  • $\begingroup$ I have viewed some related topics. What make me feel confused is the variance of LOOCV. How to compute the variance? $\endgroup$
    – Yanxi Li
    Commented May 27, 2022 at 3:23
  • $\begingroup$ @YanxiLi: "Because the train data and the test data are same." that is the collinearity I'm talking about. Think about this: Leave out samples 1 and 2, and train on 3 - 50, then predict 1. Then leave out 1 and 3 and train on 2, 4 - 50, and predict 1 again from that 2nd surrogate model. How much do the two predictions of case 1 vary? $\endgroup$ Commented May 27, 2022 at 11:04
  • $\begingroup$ @YanxiLi: how to compute variance: the part of its variance that you can compute from within the LOO-CV results can be assessed e.g. by the jack-knifing or bootstrapping procedure, see last paragraph. $\endgroup$ Commented May 27, 2022 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.