The formula I have for cross-validation error is the following: $$ \hat{R}_{CV}(f) = \frac{1}{N}\sum_{i=1}^N L\left( y_i, \hat{f}^{-k(i)}(x_i) \right) $$

where $\hat{f}^{-k(i)}$ is the model trained with $x_i$ excluded from the training set and $L$ is a loss function. Note that $k(i)$ is not necessarily injective e.g for 10-fold cross-validation $k(i)$ maps each training sample to one of the 10 folds $\{1,...,10\}$.

For variance I have the following: $$ \hat{\sigma}^2 = \frac{1}{N} \sum_{i=1}^N (L_i - \bar{L})^2 $$

where $$ L_i = L(y_i, f^{-k(i)}(x_i)) \\ \bar{L} = \frac{1}{N} \sum_{i=1}^N L_i $$

I want to use root mean squared error which is: $$ RMSE(f) = \sqrt{\frac{\sum_{i=1}^N (y_i - f(x_i))^2}{N}} $$

My problem is that I am not sure how to adapt these formulas to RMSE. To calculate the variance it seems that I need to be able to decompose my error function to a sum of $L_i$ but I cannot do that because of the square root. How can I calculate the variance if my loss function is RMSE?

  • $\begingroup$ You seem to be using leave-one-out-CV - in which case the loss for the ith fold should be the same as the absolute difference between y_i and the model's prediction. I think your formulas for the variance then just go through as normal? $\endgroup$
    – rje42
    Commented May 8, 2018 at 15:24
  • $\begingroup$ Well not really, $k(i)$ is not necessarily injective. It maps each training sample to the subset that does not include that particular point. i.e for 10-fold it maps each training sample to {1,..,10} $\endgroup$
    – Andreas G.
    Commented May 8, 2018 at 15:49
  • $\begingroup$ @AndreasG. please make that clear in the text of your question: give a definition of $k(i)$. Then I'll provide an answer to your question. $\endgroup$
    – Jim
    Commented May 8, 2018 at 20:28
  • $\begingroup$ @Jim I've made the change $\endgroup$
    – Andreas G.
    Commented May 8, 2018 at 21:35
  • $\begingroup$ @Jim No they are all of roughly the same size. It's non-injective because you can have $i \neq j$ s.t $k(i)=k(j)$ i.e two different samples mapped to the same set $\endgroup$
    – Andreas G.
    Commented May 10, 2018 at 15:31


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.