8
$\begingroup$

I am testing a neural net to predict numeric values. For that i am using a Training,Validation and Test split. I made a manual 4-Fold CV, this means i am getting 4 RMSE error, each one is the error of the i-th Fold on the test data.

How do i get global RMSE of all 4 Folds. Would it be (rmse_1 + rmse_2 + rmse_3 + rmse_4)/(number of all predictions)

$\endgroup$
7
$\begingroup$

To be correct, you should calculate the overall RMSE as $\sqrt{\frac{RMSE_1^2 + \dots + RMSE_k^2}{k}}$.

Edit: I just got from your question that it may be necessary to explain my answer a bit. The $RMSE_j$ of the instance $j$ of the cross-validation is calculated as $\sqrt{\frac{\sum_i{(y_{ij} - \hat{y}_{ij})^2}}{N_j}}$ where $\hat{y}_{ij}$ is the estimation of $y_{ij}$ and $N_j$ is the number of observations of CV instance $j$. Now the overall RMSE is something like $\sqrt{\frac{\sum_j{\frac{\sum_i{(y_{ij} - \hat{y}_{ij})^2}}{N_j}}}{k}}$ and not what you propose $\frac{\sum_j{\sqrt{\frac{\sum_i{(y_{ij} - \hat{y}_{ij})^2}}{N_j}}}}{\sum_j{N_j}}$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, but i think that first you have to undo the dividing by $k$ and the Root. Now summing the error dividing by $N_j$ and then taking the Root. Like $\sqrt{\frac{\sum_j{\sum_i{(y_i - \hat{y}_i)^2}}}{N_j}}$, where $j$ is the number of Folds and $i$ is the numer of observations. Is this thought right? $\endgroup$ – mognowich Feb 7 '14 at 16:03
  • $\begingroup$ and $N_j = \sum{N_i}$ $\endgroup$ – mognowich Feb 7 '14 at 16:06
  • $\begingroup$ I'm not sure whether I follow your thoughts, but I think you mix something up. $i$ and $j$ are indices, hence $ij$ depicts observation $i$ of fold $j$. Something like $N_i$ is meaningless, as you never have more ore less than one observation per observation ;-). $N_j$ is the number of observations of the cv fold $j$, using it outside of the $j$-sum like you do in your first comment makes no sense ... I think you should start with the first simple equation and try to follow the indices. $\endgroup$ – user1449306 Feb 7 '14 at 16:43
  • $\begingroup$ I'd appreciate a comment why $\frac{\sum_j \frac{\sum_i (y_{ij}-\hat y_{ij})^2}{N_j}}{k} = \frac{\sum_j \sum_j (y_{ij}-\hat y_{ij})^2}{\sum_j N_j}$. The latter is what I'd see as overall RMSE, the former is the square of your answer. Am I misunderstanding the term 'overall'? $\endgroup$ – Roland Jul 8 '14 at 19:07
  • $\begingroup$ What about coefficient of correlation?? $\endgroup$ – Omid Omidi Mar 22 '16 at 19:17
1
$\begingroup$

It's not a great reference, but in this notebook (look for cell starting with "Now let's compute RMSE using 10-fold x-validation") they add up the square errors (using a dot product) of all the predictions in all the cross validations, and then at the end divide by the number of predictions and square-root, i.e;

$\sqrt{\frac{1}{n} \sum_k{\sum_j{(y_{jk} - \hat{y_{jk}})^2}}}$

This makes the most sense to me, in the answer given by user1449306 the size of the folds would have an effect, which doesn't make sense?

To get to this from the list of RMSEs, they could each be squared and multiplied by the number of test points in each, then added together and divided by the total number of test points (and then squarerooted). Roland's comment is correct.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.