calculating overall error in k-fold cross validation

Question

when using k-fold cross validation i thought the overall error was equal to the mean of errors of each fold. the error being anything from MAE and RMSE to NDCG,F-measure, precision and recall. however the answer to this question tells me otherwise for RMSE. it says the following:

RMSE= $\sqrt{\frac{RMSE_1^2 + \dots + RMSE_k^2}{k}}$

why is that?

also if this one is right can you tell me what is the overall error for other metrics mentioned above?

Answer 1

• The most common way of calculating overall error for cross validation is to pool the predictions of all folds and then calculate the error of the pooled predictions.
  The basis for this are two basic assumptions of cross validation: that all the models trained and tested during the folds are treated as surrogate for testing the "whole data" model, i.e. the surrogate models are treated as equivalent to the whole data model (otherwise the cross valdidation results could not be used as error estimate for the "whole data" model). This assumption is often violated: we typically observe a slight pessimistic bias. So there's a 2nd, weaker assumption: that at least the surrogate models are equivalent to each other. This allows pooling of the results of all surrogate models.
• The answer you link instead pools the errors obtained for the individual surrogate models. Note that for this you need to make sure that you use the appropriate way of pooling for the respective figure of merit which is not always just averaging (e.g. in your example pooling RMSE as square root of average MSE).
• If the surrogate models are all tested with the same number of cases, both ways of pooling yield the same result. (And if the difference in case number is small, the difference is small or even negligible)
• If that 2nd assumption above does not hold (i.e. the surrogate models are unstable) treating the surrogate models separately is sensible.
• Nevertheless, if you do this, IMHO you should
  1. explicitly mention the pooling of the results of each surrogate model, and
  2. discuss the instability as this is an important finding in model verification/validation.

---

How to pool figures of merit

Imagine the following situation:

1. one model is tested with $2n$ (independent) test cases, resulting in a figure of merit $F_1$.
2. the same model is tested with 2 sets of test cases. The set a are the first $n$ test cases from above, the set b are the the other $n$ test cases from above, resulting in figures of merit $F_{2a}$ and $F_{2b}$.

Now, if we require that it should not matter to the test results whether a test set is broken up into disjoint subsets or not, this translates to requiring that the pooled figure of merit of $F_{2a}$ and $F_{2b}$, $F_2 = pooled (F_{2a}, F_{2b})$, should equal $F_1$:

$F_1 = F_2 = pooled (F_{2a}, F_{2b})$

More generally speaking, as long as the disjoint subsets of test cases all contain the same number of test cases, pooling the figures of merit should give the same result as pooling the test cases.

One could just as well require that the pooled figure of merit should be the same as for pooling the test cases also for unequal test (sub)set sizes - that would be asking for appropriate weighting. However, while I've in practice encountered the described pooling behavior above, I don't remember seeing the weighted pooling (instead, pooling cases avoids all further discussion).

---

Update: To clarify, of course you can calculate average fold-wise figures of merit.

• Fold-wise figures of merit are often used by optimization heuristics that take variation between fold-wise results into account (see e.g. one-standard-error heuristic).

• It may also make sense to look at fold-wise figures of merit if the models are very unstable, i.e. the surrogate models are very different, and the folds contain lots of cases so the uncertainty due to the limited number of tested cases is low.

• OTOH, if the results of the cross validation are a figure of merit that is to be used as approximation/estimate of the generalization error of a model trained on the whole data set, it is more sensible to check that the training procedure yields stable cross validation surrogate models and then pool the predictions.
  As I said above, whether the figure of merit for the pooled predictions is then calculated by appropriately aggregating fold-wise figures of merit or by pooling the predictions doesn't matter.