When using repeated cross-validation for evaluation, should I aggregate the results for each cross-validation?

When aggregating, a 10-times repeated 10-fold, will provide 10 values for each measure like AUC. (Therefore, k=10 for my t-test with k-1 degrees of freedom when comparing algorithms.)

When not-aggregating, a 10-times repeated 10-fold will provide 100 values, as each fold is considered as a single run.

WEKA does not aggregate. However, Han writes in his data mining book (p370) that each CV should be aggregated.

I personally am a fan of not-aggregating: a 10 x 10-fold CV produces 100 models, so one should use 100 results for comparison. Some of my colleagues disagree, though.

Addition: I am doing the repeated CV for:

  • Comparing two algorithms with a paired t-test

  • Getting a variance estimate for my model performance

  • $\begingroup$ Practically how do you compare 100 numbers? The brains fits 7 at a time in any case. You'll end up aggregating in the end one way or another, explicitly or implictly. It's better to do explicitly $\endgroup$ – Aksakal Oct 13 '16 at 15:15

As mentioned above WEKA uses the 100 results of the single folds as input for the (corrected resampled) t-test.

According to a mailing list post, aggregating the results for each cross-validation run would lead to an increased Type I error rate (i.e., not existing significant differences are detected between algorithms).

Hence, each repetition of the cross-validation should NOT be aggregated before the t-test.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.