Paired t-test in testing difference between two classifiers

Question

While reading the paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", I am confused at Section 3.3 when the authors claim that the paired t-test is not suitable to use to compare two classifiers $A$ and $B$, because the observed proportion of test examples misclassified $p_A$ and $p_B$ by $A$ and $B$ during trial $i$ are not independent. However, according to my understanding, the paired t-test assumes that the samples are paired, thus dependent.

What I reckon is that the test mentioned in the paper is the unpaired t-test. However, I think there is no motivation to consider unpaired t-test to compare two classifiers.

Thank you in advance!

Answer 1

We use a paired t-test when we have two measurements on the same sampling unit (dependent observations) AND when sampling units are independent of one another. Observations are paired, not samples.

In the situation in Section 3.3, the sampling units are NOT independent of one another because sampling units are created artificially by resampling the sample dataset (S). A paired t-test is therefore not appropriate.

The test mentioned in the paper uses a formula based on differences (p(i)), so it is referring to a paired t-test.