11
$\begingroup$

I am testing a neural net to predict numeric values. For that i am using a Training,Validation and Test split. I made a manual 4-Fold CV, this means i am getting 4 RMSE error, each one is the error of the i-th Fold on the test data.

How do i get global RMSE of all 4 Folds. Would it be (rmse_1 + rmse_2 + rmse_3 + rmse_4)/(number of all predictions)

Improve this question
$\endgroup$
1

3 Answers 3

Reset to default
10
$\begingroup$

To be correct, you should calculate the overall RMSE as $\sqrt{\frac{RMSE_1^2 + \dots + RMSE_k^2}{k}}$.

Edit: I just got from your question that it may be necessary to explain my answer a bit. The $RMSE_j$ of the instance $j$ of the cross-validation is calculated as $\sqrt{\frac{\sum_i{(y_{ij} - \hat{y}_{ij})^2}}{N_j}}$ where $\hat{y}_{ij}$ is the estimation of $y_{ij}$ and $N_j$ is the number of observations of CV instance $j$. Now the overall RMSE is something like $\sqrt{\frac{\sum_j{\frac{\sum_i{(y_{ij} - \hat{y}_{ij})^2}}{N_j}}}{k}}$ and not what you propose $\frac{\sum_j{\sqrt{\frac{\sum_i{(y_{ij} - \hat{y}_{ij})^2}}{N_j}}}}{\sum_j{N_j}}$.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks, but i think that first you have to undo the dividing by $k$ and the Root. Now summing the error dividing by $N_j$ and then taking the Root. Like $\sqrt{\frac{\sum_j{\sum_i{(y_i - \hat{y}_i)^2}}}{N_j}}$, where $j$ is the number of Folds and $i$ is the numer of observations. Is this thought right? $\endgroup$
    – mognowich
    Feb 7, 2014 at 16:03
  • $\begingroup$ and $N_j = \sum{N_i}$ $\endgroup$
    – mognowich
    Feb 7, 2014 at 16:06
  • $\begingroup$ I'm not sure whether I follow your thoughts, but I think you mix something up. $i$ and $j$ are indices, hence $ij$ depicts observation $i$ of fold $j$. Something like $N_i$ is meaningless, as you never have more ore less than one observation per observation ;-). $N_j$ is the number of observations of the cv fold $j$, using it outside of the $j$-sum like you do in your first comment makes no sense ... I think you should start with the first simple equation and try to follow the indices. $\endgroup$
    – user1449306
    Feb 7, 2014 at 16:43
  • $\begingroup$ I'd appreciate a comment why $\frac{\sum_j \frac{\sum_i (y_{ij}-\hat y_{ij})^2}{N_j}}{k} = \frac{\sum_j \sum_j (y_{ij}-\hat y_{ij})^2}{\sum_j N_j}$. The latter is what I'd see as overall RMSE, the former is the square of your answer. Am I misunderstanding the term 'overall'? $\endgroup$
    – Roland
    Jul 8, 2014 at 19:07
  • $\begingroup$ What about coefficient of correlation?? $\endgroup$
    – Omid Omidi
    Mar 22, 2016 at 19:17
1
$\begingroup$

It's not a great reference, but in this notebook (look for cell starting with "Now let's compute RMSE using 10-fold x-validation") they add up the square errors (using a dot product) of all the predictions in all the cross validations, and then at the end divide by the number of predictions and square-root, i.e;

$\sqrt{\frac{1}{n} \sum_k{\sum_j{(y_{jk} - \hat{y_{jk}})^2}}}$

This makes the most sense to me, in the answer given by user1449306 the size of the folds would have an effect, which doesn't make sense?

To get to this from the list of RMSEs, they could each be squared and multiplied by the number of test points in each, then added together and divided by the total number of test points (and then squarerooted). Roland's comment is correct.

Improve this answer
$\endgroup$
0
$\begingroup$

Here's my take. First we square the values, then we multiply with the counts, then we add them all up, divide by total count and take the square root. In pseudocode (Python), this could be implemented like

# Overall RMSE from list of rmses
new_list=[x*x for x in fold_wise_rmse]
counts  =[len(x) for x in val_sets]
nldotcounts=np.dot(new_list,counts)
overall_rmse=np.sqrt(nldotcounts/sum(counts))

where fold_wise_rmse and val_sets are both list structures

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.