# Paired t-test in testing difference between two classifiers

While reading the paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", I am confused at Section 3.3 when the authors claim that the paired t-test is not suitable to use to compare two classifiers $A$ and $B$, because the observed proportion of test examples misclassified $p_A$ and $p_B$ by $A$ and $B$ during trial $i$ are not independent. However, according to my understanding, the paired t-test assumes that the samples are paired, thus dependent.

What I reckon is that the test mentioned in the paper is the unpaired t-test. However, I think there is no motivation to consider unpaired t-test to compare two classifiers.

Thank you in advance!

• The part I agree with the paper is that $p^{(i)}$'s are not independent, because the test sets overlap.The confusing part is "... because $p_A$ and $p_B$ are not independent". I am confused since because they are not independent, paired t-test is used in the first place. – little_monster Apr 11 '17 at 23:54