Comparing two models using Repeated K-fold Cross Validation

Question

I am creating two separate classification models. In order to estimate the performance, I have been using 10-fold cross validation repeated 5 times for each model, resulting in 50 different performance metrics for each model. If I wanted to say for sure that one model's performance was better than another, and that this result is statistically significant, can I calculate the mean of the 50 different runs for each model and then perform a t-test? Are there assumptions being violated if I do this?

Answer 1

Update:

I've been doing some reading the past few days and found a few answers. According to Dietterich's paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", standard t-tests have a high type I error (a significant difference is found between the models even when there isn't actually one) when used on cross validated results. According to Nadeau and Bengio's paper "Inference for the Generalization Error", this is due to the underestimation of the variance the samples caused by them not being truly independent - the test data is re-used in other folds.

To get around this, I found another paper "Evaluating the replicability of significance tests for comparing learning algorithms" that recommends the use of a "corrected" version of the t-test for repeated k-fold cross validation (which was used in my original models). The corrected version is:

$$t= \frac{\frac{1}{kr}\sum_{i=1}^{k}\sum_{j=1}^{r} x_{ij}}{\sqrt{(\frac{1}{kr} + \frac{n_2}{n_1})\hat{\sigma}^2}}$$

Where $k$ is a given fold, $r$ is the number of repeats, $x_{ij}$ is the difference between models in a given run $j$ and fold $i$. The numerator is an estimate for the mean, while the variance can be estimated as $\hat{\sigma}^2=\frac{1}{kr-1}\sum_{i=1}^{k}\sum_{j=1}^{r}(x_{ij}-m).^2$

(EDIT by @Firebug: $n_1$ is the number of instances used for training, $n_2$ is the number of instances used for testing)

This version seems to be more consistent and replicable in experiments according to the author, so it seems like a safer solution than a naive t-test.