The formula I have for cross-validation error is the following: $$ \hat{R}_{CV}(f) = \frac{1}{N}\sum_{i=1}^N L\left( y_i, \hat{f}^{-k(i)}(x_i) \right) $$

where $\hat{f}^{-k(i)}$ is the model trained with $x_i$ excluded from the training set and $L$ is a loss function. Note that $k(i)$ is not necessarily injective e.g for 10-fold cross-validation $k(i)$ maps each training sample to one of the 10 folds $\{1,...,10\}$.

For variance I have the following: $$ \hat{\sigma}^2 = \frac{1}{N} \sum_{i=1}^N (L_i - \bar{L})^2 $$

where $$ L_i = L(y_i, f^{-k(i)}(x_i)) \\ \bar{L} = \frac{1}{N} \sum_{i=1}^N L_i $$

I want to use root mean squared error which is: $$ RMSE(f) = \sqrt{\frac{\sum_{i=1}^N (y_i - f(x_i))^2}{N}} $$

My problem is that I am not sure how to adapt these formulas to RMSE. To calculate the variance it seems that I need to be able to decompose my error function to a sum of $L_i$ but I cannot do that because of the square root. How can I calculate the variance if my loss function is RMSE?

  • $\begingroup$ You seem to be using leave-one-out-CV - in which case the loss for the ith fold should be the same as the absolute difference between y_i and the model's prediction. I think your formulas for the variance then just go through as normal? $\endgroup$ – rje42 May 8 '18 at 15:24
  • $\begingroup$ Well not really, $k(i)$ is not necessarily injective. It maps each training sample to the subset that does not include that particular point. i.e for 10-fold it maps each training sample to {1,..,10} $\endgroup$ – Andreas G. May 8 '18 at 15:49
  • $\begingroup$ @AndreasG. please make that clear in the text of your question: give a definition of $k(i)$. Then I'll provide an answer to your question. $\endgroup$ – Jim May 8 '18 at 20:28
  • $\begingroup$ @Jim I've made the change $\endgroup$ – Andreas G. May 8 '18 at 21:35
  • $\begingroup$ @Jim No they are all of roughly the same size. It's non-injective because you can have $i \neq j$ s.t $k(i)=k(j)$ i.e two different samples mapped to the same set $\endgroup$ – Andreas G. May 10 '18 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.