when using k-fold cross validation i thought the overall error was equal to the mean of errors of each fold. the error being anything from MAE and RMSE to NDCG,F-measure, precision and recall. however the answer to this question tells me otherwise for RMSE. it says the following:

RMSE= $\sqrt{\frac{RMSE_1^2 + \dots + RMSE_k^2}{k}}$

why is that?

also if this one is right can you tell me what is the overall error for other metrics mentioned above?

  • The most common way of calculating overall error for cross validation is to pool the predictions of all folds and then calculate the error of the pooled predictions.
    The basis for this are two basic assumptions of cross validation: that all the models trained and tested during the folds are treated as surrogate for testing the "whole data" model, i.e. the surrogate models are treated as equivalent to the whole data model (otherwise the cross valdidation results could not be used as error estimate for the "whole data" model). This assumption is often violated: we typically observe a slight pessimistic bias. So there's a 2nd, weaker assumption: that at least the surrogate models are equivalent to each other. This allows pooling of the results of all surrogate models.
  • The answer you link instead pools the errors obtained for the individual surrogate models. Note that for this you need to make sure that you use the appropriate way of pooling for the respective figure of merit which is not always just averaging (e.g. in your example pooling RMSE as square root of average MSE).
  • If the surrogate models are all tested with the same number of cases, both ways of pooling yield the same result. (And if the difference in case number is small, the difference is small or even negligible)
  • If that 2nd assumption above does not hold (i.e. the surrogate models are unstable) treating the surrogate models separately is sensible.
  • Nevertheless, if you do this, IMHO you should
    1. explicitly mention the pooling of the results of each surrogate model, and
    2. discuss the instability as this is an important finding in model verification/validation.

How to pool figures of merit

Imagine the following situation:

  1. one model is tested with $2n$ (independent) test cases, resulting in a figure of merit $F_1$.
  2. the same model is tested with 2 sets of test cases. The set a are the first $n$ test cases from above, the set b are the the other $n$ test cases from above, resulting in figures of merit $F_{2a}$ and $F_{2b}$.

Now, if we require that it should not matter to the test results whether a test set is broken up into disjoint subsets or not, this translates to requiring that the pooled figure of merit of $F_{2a}$ and $F_{2b}$, $F_2 = pooled (F_{2a}, F_{2b})$, should equal $F_1$:

$F_1 = F_2 = pooled (F_{2a}, F_{2b})$

More generally speaking, as long as the disjoint subsets of test cases all contain the same number of test cases, pooling the figures of merit should give the same result as pooling the test cases.

One could just as well require that the pooled figure of merit should be the same as for pooling the test cases also for unequal test (sub)set sizes - that would be asking for appropriate weighting. However, while I've in practice encountered the described pooling behavior above, I don't remember seeing the weighted pooling (instead, pooling cases avoids all further discussion).

  • $\begingroup$ thank you. this answer clarifies my confusion on how to calculate overall MAE and RMSE. but it's a different case with NDCG, recall, precision and f-measure. what about those? are they calculated as the mean of accuracy of each fold? $\endgroup$ – Zahra Moradi Jan 8 at 15:36
  • $\begingroup$ @ZahraMoradi: well, nothing really changes. But you need to take care to use the right pooling function depending on the figure of merit you look at. I edited the answer to clarify this. $\endgroup$ – cbeleites supports Monica Jan 9 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.