How to calculate CV performance?

Question

There seem to exist two ways of calculating the $k$-fold cross-validation performance of a model.

1. For each fold, evaluate the performance measure (e.g. MSE) on the hold-out observations. Then, calculate the average of these $k$ values.

2. For each fold, calculate the predictions on the hold-out set. Then, calculate the single performance score from all pooled predictions together.

In my opinion, the first method is easier to implement in parallelized calculations, while the second one somehow feels more stable for small hold-out sets. Take e.g. a leave-one-out CV with R-squared as performance measure. There, only Option 2 is possible.

Are both options considered to be proper? Are there any good references or hints about this?

Answer 1

As you said, the second method is preferred for small hold out sets (or large values of $k$ compared to the $n$ of your data-set if you will). At the extreme end, you will almost always use the second for leave one out. (Think twice before scoring on $R^2$ though.)

The first has the advantage that you can use the multiple performance metrics, one per fold, for t-tests and such. (But you must correct for the pseudo-replication with corrected resampled standard errors.)

Both methods are easy to parallelize if you have enough RAM to run as many instances of the algorithm in parallel as you have CPU cores. The only difference between both methods is the time at which the performance metrics are computed. That part is a very small effort for your computer anyway.