5
$\begingroup$

I am creating two separate classification models. In order to estimate the performance, I have been using 10-fold cross validation repeated 5 times for each model, resulting in 50 different performance metrics for each model. If I wanted to say for sure that one model's performance was better than another, and that this result is statistically significant, can I calculate the mean of the 50 different runs for each model and then perform a t-test? Are there assumptions being violated if I do this?

$\endgroup$
1
  • 1
    $\begingroup$ Better do bootstrap estimation of the mean performance and its standard errors. $\endgroup$ Jul 23, 2021 at 22:25

1 Answer 1

3
$\begingroup$

Update:

I've been doing some reading the past few days and found a few answers. According to Dietterich's paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", standard t-tests have a high type I error (a significant difference is found between the models even when there isn't actually one) when used on cross validated results. According to Nadeau and Bengio's paper "Inference for the Generalization Error", this is due to the underestimation of the variance the samples caused by them not being truly independent - the test data is re-used in other folds.

To get around this, I found another paper "Evaluating the replicability of significance tests for comparing learning algorithms" that recommends the use of a "corrected" version of the t-test for repeated k-fold cross validation (which was used in my original models). The corrected version is:

$$t= \frac{\frac{1}{kr}\sum_{i=1}^{k}\sum_{j=1}^{r} x_{ij}}{\sqrt{(\frac{1}{kr} + \frac{n_2}{n_1})\hat{\sigma}^2}}$$

Where $k$ is a given fold, $r$ is the number of repeats, $x_{ij}$ is the difference between models in a given run $j$ and fold $i$. The numerator is an estimate for the mean, while the variance can be estimated as $\hat{\sigma}^2=\frac{1}{kr-1}\sum_{i=1}^{k}\sum_{j=1}^{r}(x_{ij}-m).^2$

(EDIT by @Firebug: $n_1$ is the number of instances used for training, $n_2$ is the number of instances used for testing)

This version seems to be more consistent and replicable in experiments according to the author, so it seems like a safer solution than a naive t-test.

$\endgroup$
3
  • 1
    $\begingroup$ What are n1 and n2? $\endgroup$
    – HaneenSu
    Oct 28, 2022 at 6:37
  • $\begingroup$ @HaneenSu $n_1$ is the number of instances used for training, $n_2$ is the number of instances used for testing $\endgroup$
    – Firebug
    Feb 28, 2023 at 9:48
  • $\begingroup$ I've added this info to the answer, if someone disagrees feel free to revert it $\endgroup$
    – Firebug
    Mar 1, 2023 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.