when using k-fold cross validation i thought the overall error was equal to the mean of errors of each fold. the error being anything from MAE and RMSE to NDCG,F-measure, precision and recall. however the answer to this question tells me otherwise for RMSE. it says the following:

RMSE= $\sqrt{\frac{RMSE_1^2 + \dots + RMSE_k^2}{k}}$

why is that?

also if this one is right can you tell me what is the overall error for other metrics mentioned above?


1 Answer 1

  • The most common way of calculating overall error for cross validation is to pool the predictions of all folds and then calculate the error of the pooled predictions.
    The basis for this are two basic assumptions of cross validation: that all the models trained and tested during the folds are treated as surrogate for testing the "whole data" model, i.e. the surrogate models are treated as equivalent to the whole data model (otherwise the cross valdidation results could not be used as error estimate for the "whole data" model). This assumption is often violated: we typically observe a slight pessimistic bias. So there's a 2nd, weaker assumption: that at least the surrogate models are equivalent to each other. This allows pooling of the results of all surrogate models.
  • The answer you link instead pools the errors obtained for the individual surrogate models. Note that for this you need to make sure that you use the appropriate way of pooling for the respective figure of merit which is not always just averaging (e.g. in your example pooling RMSE as square root of average MSE).
  • If the surrogate models are all tested with the same number of cases, both ways of pooling yield the same result. (And if the difference in case number is small, the difference is small or even negligible)
  • If that 2nd assumption above does not hold (i.e. the surrogate models are unstable) treating the surrogate models separately is sensible.
  • Nevertheless, if you do this, IMHO you should
    1. explicitly mention the pooling of the results of each surrogate model, and
    2. discuss the instability as this is an important finding in model verification/validation.

How to pool figures of merit

Imagine the following situation:

  1. one model is tested with $2n$ (independent) test cases, resulting in a figure of merit $F_1$.
  2. the same model is tested with 2 sets of test cases. The set a are the first $n$ test cases from above, the set b are the the other $n$ test cases from above, resulting in figures of merit $F_{2a}$ and $F_{2b}$.

Now, if we require that it should not matter to the test results whether a test set is broken up into disjoint subsets or not, this translates to requiring that the pooled figure of merit of $F_{2a}$ and $F_{2b}$, $F_2 = pooled (F_{2a}, F_{2b})$, should equal $F_1$:

$F_1 = F_2 = pooled (F_{2a}, F_{2b})$

More generally speaking, as long as the disjoint subsets of test cases all contain the same number of test cases, pooling the figures of merit should give the same result as pooling the test cases.

One could just as well require that the pooled figure of merit should be the same as for pooling the test cases also for unequal test (sub)set sizes - that would be asking for appropriate weighting. However, while I've in practice encountered the described pooling behavior above, I don't remember seeing the weighted pooling (instead, pooling cases avoids all further discussion).

Update: To clarify, of course you can calculate average fold-wise figures of merit.

  • Fold-wise figures of merit are often used by optimization heuristics that take variation between fold-wise results into account (see e.g. one-standard-error heuristic).

  • It may also make sense to look at fold-wise figures of merit if the models are very unstable, i.e. the surrogate models are very different, and the folds contain lots of cases so the uncertainty due to the limited number of tested cases is low.

  • OTOH, if the results of the cross validation are a figure of merit that is to be used as approximation/estimate of the generalization error of a model trained on the whole data set, it is more sensible to check that the training procedure yields stable cross validation surrogate models and then pool the predictions.
    As I said above, whether the figure of merit for the pooled predictions is then calculated by appropriately aggregating fold-wise figures of merit or by pooling the predictions doesn't matter.

  • $\begingroup$ thank you. this answer clarifies my confusion on how to calculate overall MAE and RMSE. but it's a different case with NDCG, recall, precision and f-measure. what about those? are they calculated as the mean of accuracy of each fold? $\endgroup$
    – Zahra
    Jan 8, 2019 at 15:36
  • $\begingroup$ @ZahraMoradi: well, nothing really changes. But you need to take care to use the right pooling function depending on the figure of merit you look at. I edited the answer to clarify this. $\endgroup$ Jan 9, 2019 at 12:02
  • $\begingroup$ Doesn't this differ from what scikit-learn does? That seems to calculate one score (error metric) per fold, then average the k error metrics. See scikit-learn.org/stable/modules/cross_validation.html and the source code at github.com/scikit-learn/scikit-learn/blob/b194674c4/sklearn/… $\endgroup$
    – Will
    Feb 18, 2020 at 17:08
  • 1
    $\begingroup$ @Will: indeed if you look around you'll see a whole lot of different pooling strategies employed, which make more or less sense depending on the particular situation. I updated the answer accordingly. However, calculating the figure of merit for all N cases pooled (i.e. pooling predictions across folds) is e.g. the formula given by The Elements of Statistical Learning. It may very well be that the scikit developers decided to go for fold-wise measures instead because that's handy for optimization and the pooled results can still be calculated from the fold-wise at the very least approximately. $\endgroup$ Feb 19, 2020 at 22:18
  • 1
    $\begingroup$ FYI @cbeleitessupportsMonica: page 61 of Elements of Statistical Learning describes crossvalidation differently than the formula later on: “ The learning method is fit—for a range of values of the complexity parameter—to nine-tenths of the data, and the prediction error is computed on the remaining one-tenth. This is done in turn for each one-tenth of the data, and the ten prediction error estimates are averaged. From this we obtain an estimated prediction error curve as a function of the complexity parameter.” So, even they are ambivalent. But, the math works out to be the same anyway. $\endgroup$
    – Will
    Feb 25, 2020 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.