When using repeated cross-validation for evaluation, should I aggregate the results for each cross-validation?
When aggregating, a 10-times repeated 10-fold, will provide 10 values for each measure like AUC. (Therefore, k=10 for my t-test with k-1 degrees of freedom when comparing algorithms.)
When not-aggregating, a 10-times repeated 10-fold will provide 100 values, as each fold is considered as a single run.
I personally am a fan of not-aggregating: a 10 x 10-fold CV produces 100 models, so one should use 100 results for comparison. Some of my colleagues disagree, though.
Addition: I am doing the repeated CV for:
Comparing two algorithms with a paired t-test
Getting a variance estimate for my model performance