Return to Answer

Update:

I've been doing some reading the past few days and found a few answers. According to Dietterich's paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", standard t-tests have a high type I error (a significant difference is found between the models even when there isn't actually one) when used on cross validated results. According to Nadeau and Bengio's paper "Inference for the Generalization Error", this is due to the underestimation of the variance the samples caused by them not being truly independent - the test data is re-used in other folds.

To get around this, I found another paper "Evaluating the replicability of significance tests for comparing learning algorithms" that recommends the use of a "corrected" version of the t-test for repeated k-fold cross validation (which was used in my original models). The corrected version is:

$$t= \frac{\frac{1}{kr}\sum_{i=1}^{k}\sum_{j=1}^{r} x_{ij}}{\sqrt{(\frac{1}{kr} + \frac{n_2}{n_1})\hat{\sigma}^2}}$$

Where $k$ is a given fold, $r$ is the number of repeats, $x_{ij}$ is the difference between models in a given run $j$ and fold $i$. The numerator is an estimate for the mean, while the variance can be estimated as $\hat{\sigma}^2=\frac{1}{kr-1}\sum_{i=1}^{k}\sum_{j=1}^{r}(x_{ij}-m).^2$

This version seems to be more consistent and replicable in experiments according to the author, so it seems like a safer solution than a naive t-test.

Update:

I've been doing some reading the past few days and found a few answers. According to Dietterich's paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", standard t-tests have a high type I error (a significant difference is found between the models even when there isn't actually one) when used on cross validated results. According to Nadeau and Bengio's paper "Inference for the Generalization Error", this is due to the underestimation of the variance the samples caused by them not being truly independent - the test data is re-used in other folds.

To get around this, I found another paper "Evaluating the replicability of significance tests for comparing learning algorithms" that recommends the use of a "corrected" version of the t-test for repeated k-fold cross validation (which was used in my original models). The corrected version is:

$$t= \frac{\frac{1}{kr}\sum_{i=1}^{k}\sum_{j=1}^{r} x_{ij}}{\sqrt{(\frac{1}{kr} + \frac{n_2}{n_1})\hat{\sigma}^2}}$$

Where $k$ is a given fold, $r$ is the number of repeats, $x_{ij}$ is the difference between models in a given run $j$ and fold $i$. The numerator is an estimate for the mean, while the variance can be estimated as $\hat{\sigma}^2=\frac{1}{kr-1}\sum_{i=1}^{k}\sum_{j=1}^{r}(x_{ij}-m).^2$

(EDIT by @Firebug: $n_1$ is the number of instances used for training, $n_2$ is the number of instances used for testing)

This version seems to be more consistent and replicable in experiments according to the author, so it seems like a safer solution than a naive t-test.

Update:

I've been doing some reading the past few days and found a few answers. According to Dietterich's paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", standard t-tests have a high type I error (a significant difference is found between the models even when there isn't actually one) when used on cross validated results. According to Nadeau and Bengio's paper "Inference for the Generalization Error", this is due to the underestimation of the variance the samples caused by them not being truly independent - the test data is re-used in other folds.

To get around this, I found another paper "Evaluating the replicability of significance tests for comparing learning algorithms" that recommends the use of a "corrected" version of the t-test for repeated k-fold cross validation (which was used in my original models). The corrected version is:

$$t= \frac{\frac{kr}{n}\sum_{i=1}^{k}\sum_{j=1}^{r} x_{ij}}{\sqrt{(\frac{1}{kr} + \frac{n_2}{n_1})\hat{\sigma}^2}}$$

Where $k$ is a given fold, $r$ is the number of repeats, $x_{ij}$ is the difference between models in a given run $j$ and fold $i$. The numerator is an estimate for the mean, while the variance can be estimated as $\hat{\sigma}^2=\frac{1}{kr-1}\sum_{i=1}^{k}\sum_{j=1}^{r}(x_{ij}-m).^2$

This version seems to be more consistent and replicable in experiments according to the author, so it seems like a safer solution than a naive t-test.

Update:

I've been doing some reading the past few days and found a few answers. According to Dietterich's paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", standard t-tests have a high type I error (a significant difference is found between the models even when there isn't actually one) when used on cross validated results. According to Nadeau and Bengio's paper "Inference for the Generalization Error", this is due to the underestimation of the variance the samples caused by them not being truly independent - the test data is re-used in other folds.

To get around this, I found another paper "Evaluating the replicability of significance tests for comparing learning algorithms" that recommends the use of a "corrected" version of the t-test for repeated k-fold cross validation (which was used in my original models). The corrected version is:

$$t= \frac{\frac{1}{kr}\sum_{i=1}^{k}\sum_{j=1}^{r} x_{ij}}{\sqrt{(\frac{1}{kr} + \frac{n_2}{n_1})\hat{\sigma}^2}}$$

Where $k$ is a given fold, $r$ is the number of repeats, $x_{ij}$ is the difference between models in a given run $j$ and fold $i$. The numerator is an estimate for the mean, while the variance can be estimated as $\hat{\sigma}^2=\frac{1}{kr-1}\sum_{i=1}^{k}\sum_{j=1}^{r}(x_{ij}-m).^2$

This version seems to be more consistent and replicable in experiments according to the author, so it seems like a safer solution than a naive t-test.

Update:

I've been doing some reading the past few days and found a few answers. According to Dietterich's paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", standard t-tests have a high type I error (a significant difference is found between the models even when there isn't actually one) when used on cross validated results. According to Nadeau and Bengio's paper "Inference for the Generalization Error", this is due to the underestimation of the variance the samples are not independent - the test data is re-used in other folds.

To get around this, I found another paper "Evaluating the replicability of significance tests for comparing learning algorithms" that recommends the use of a "corrected" version of the t-test for repeated k-fold cross validation (which was used in my original models). The corrected version is:

$$t= \frac{\frac{1}{n}\sum_{j=1}^{n} x_{j}}{\sqrt{(\frac{1}{n} + \frac{n_2}{n_1})\hat{\sigma}^2}}$$

Where $j$ is a given run, $n_1$ are the number of instances used for training, $n_2$ are the instances used for testing, $a_j$ and $b_j$ are the accuracy (or any other statistic) of each model being compared and $x_j = a_j - b_j$, and $\hat{\sigma}^2$ is the estimate of the variance of the $n$ differences.

This version seems to be more consistent and replicable in experiments according to the author, so it seems like a safer solution than a naive t-test.

Update:

I've been doing some reading the past few days and found a few answers. According to Dietterich's paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", standard t-tests have a high type I error (a significant difference is found between the models even when there isn't actually one) when used on cross validated results. According to Nadeau and Bengio's paper "Inference for the Generalization Error", this is due to the underestimation of the variance the samples caused by them not being truly independent - the test data is re-used in other folds.

To get around this, I found another paper "Evaluating the replicability of significance tests for comparing learning algorithms" that recommends the use of a "corrected" version of the t-test for repeated k-fold cross validation (which was used in my original models). The corrected version is:

$$t= \frac{\frac{kr}{n}\sum_{i=1}^{k}\sum_{j=1}^{r} x_{ij}}{\sqrt{(\frac{1}{kr} + \frac{n_2}{n_1})\hat{\sigma}^2}}$$

Where $k$ is a given fold, $r$ is the number of repeats, $x_{ij}$ is the difference between models in a given run $j$ and fold $i$. The numerator is an estimate for the mean, while the variance can be estimated as $\hat{\sigma}^2=\frac{1}{kr-1}\sum_{i=1}^{k}\sum_{j=1}^{r}(x_{ij}-m).^2$

This version seems to be more consistent and replicable in experiments according to the author, so it seems like a safer solution than a naive t-test.