5
$\begingroup$

When performing a k-fold cross-validation, the standard method is to evaluate the error on each test fold and take the arithmetic average of the statistic. What if the test statistic is sensitive to sample size? Would it be statistically defensible to join the predicted values from the k test folds and evaluate the test statistic on this joined data (which will be the same size as the original data)?


A simple illustrative example:

We want to perform leave-one-out cross-validation on a series of data. We will evaluate the performance using the Nash-Sutcliffe Efficiency (NSE), which is a statistic frequently used in hydrology:

$$NSE = 1-\frac{\sum_{t=1}^{T}(Q_o^t-Q_m^t)^2}{\sum_{t=1}^{T}(Q_o^t-\bar{Q}_o)^2}$$

Q is the discharge (our dependent variable), $t\in[1,2,...,T]$ are the timesteps, the $_o$ subscript indicates observed data and the $_m$ subscript indicates modelled data.

The denominator is just the variance of the observed data. We are doing leave-one-out cross-validation, so $T=1$, and thus the statistic is undefined. However, I could keep track of all the $Q_m$ and compute the NSE on the full time series.


Is this ever done? Are there any theoretical problems with this?

$\endgroup$

1 Answer 1

3
$\begingroup$

It's perfectly fine to pool the individual predictions across all surrogate models.


the standard method is to evaluate the error on each test fold and take the arithmetic average of the statistic

I don't think so.

Some software implementations do this by default, others by default pool the predictions from all surrogate models.


Key assumptions of cross validation are that the surrogate models are equivalent (sufficiently similar) to the model built on the whole data set, so that they can actually be used as a surrogate for estimating generalization error.

A second, weaker assumption recognizes that frequently the surrogate models have on average somewhat worse performance since they are built on fewer training cases (explaining a small pessimistic bias of cross validation estimates of generalization error).

Since the cases are anyways assumed to come from the same distribution, this allows to pool the cross validation results across the surrogate models.

Note that this approach yields results that are rather independent on the choice of $k$, which is IMHO a preferrable property since $k$ is in no way related to the predictive performance of the model built on the whole data set.


For some figures of merit, the arithmetic mean is a "natural" way of averaging since it doesn't matter how you "pre-pool" your individual predictions, the result will be the same for equal fold sizes. That is e.g. the case for the mean squared error MSE.

For other figures of merit, e.g. the root mean squared error RMSE, this is not true and I'd advise some careful thought which way of pooling the per-fold results is then appropriate. The same holds for your NSE calculations.


I'd advise against leave-one-out cross validation in general since it leads to collinearity between surrogate model and tested case and thus does not allow to distinguish between model instability and case-to-case variance in the prediction error.

Since the fold sizes in k-fold cross validation should differ by not more than 1 case, I'd expect the difference between both ways of averaging usually to be small compared to the total uncertainty (due to model training instability and due to case-to-case variance) on the estimates.

For the particular case of the NSE, the variance in the denominator asks for sufficiently large folds if per-fold NSEs are to be meaningful.

$\endgroup$
1
  • $\begingroup$ Thank you for this insightful answer. I follow up to "Since the fold sizes in k-fold cross validation should differ by not more than 1 case, I'd expect the difference between both ways of averaging usually to be small". For many figures of merit I think the logic here does not hold. For example, for RMSE it doesn't hold because $\frac{1}{k}\sum_{j=1}^k\sqrt{\frac{1}{m}\sum_{i=1}^m(x_{ij}-\hat{x_{ij}})^2} \neq \sqrt{\frac{1}{k}\frac{1}{m}\sum_{j=1}^k\sum_{i=1}^m(x_{ij}-\hat{x_{ij}})^2}$ $\endgroup$ Commented Feb 23 at 0:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.