Independent Samples T-test vs Paired Samples T-test for hypothesis testing (machine learning algorithms comparison)

Question

The scenario is as follows: I have 1 dataset and I want to compare the performance of different algorithms. Let's say A, B, C.

I train each algorithm a 1000 times in one training loop. At the beginning of each loop the dataset will be split into a train-test split (random split). At the end I will have 3 distributions, each containing 1000 scores (e.g., AUC scores).

Question: Which T-test should I perform? A independent samples t-test or a paired-samples t-test. I'm struggling with the difference in independence, and also an indepedent samples-test assumes that variance between is roughly the same.

Could someone please help me?

Gr

Answer 1

You need to clearly state your hypothesis. For some reason, this is a step that is commonly overlooked by people who wish to make some kind of statistical inference, and instead the focus is on "what test do I use" as if all that anyone needs to do is crunch numbers through some statistical procedure to get at the answer. The choice of hypothesis directly informs the appropriate testing procedure.

In your case, I don't know if what you want is to make an inference about the mean scores, the median scores, or the shape of the distribution of the scores, or some other distributional property of the scores; you have not specified this.

Moreover, you have not specified whether you are interested in an omnibus test or a pairwise comparison between algorithms.

Both of the above are necessary to specify, in the sense that not doing so means you cannot choose an appropriate test.

Furthermore, there are assumptions about the parameter(s) for which you wish to make such an inference. For instance, if the inference is on the mean scores, we might reasonably say that the mean will be approximately normally distributed, but this may not be the case. Extreme deviations from normality would change your choice of test statistic.

I was going to provide examples of specific tests; e.g., a one-way ANOVA, or a pairwise nonparametric test like Mann-Whitney, but realized that such information would be liable to be misused in the absence of an inadequate description of your hypothesis.