While reading the paper "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms", I am confused at Section 3.3 when the authors claim that the paired t-test is not suitable to use to compare two classifiers $A$ and $B$, because the observed proportion of test examples misclassified $p_A$ and $p_B$ by $A$ and $B$ during trial $i$ are not independent. However, according to my understanding, the paired t-test assumes that the samples are paired, thus dependent.

What I reckon is that the test mentioned in the paper is the unpaired t-test. However, I think there is no motivation to consider unpaired t-test to compare two classifiers.

Thank you in advance!


1 Answer 1


We use a paired t-test when we have two measurements on the same sampling unit (dependent observations) AND when sampling units are independent of one another. Observations are paired, not samples.

In the situation in Section 3.3, the sampling units are NOT independent of one another because sampling units are created artificially by resampling the sample dataset (S). A paired t-test is therefore not appropriate.

The test mentioned in the paper uses a formula based on differences (p(i)), so it is referring to a paired t-test.

  • $\begingroup$ Thank you for your input. However, I reckon you were mistaken between random sampling and independent/dependent observations. $\endgroup$ Commented Apr 11, 2017 at 23:37
  • 1
    $\begingroup$ The part I agree with the paper is that $p^{(i)}$'s are not independent, because the test sets overlap.The confusing part is "... because $p_A$ and $p_B$ are not independent". I am confused since because they are not independent, paired t-test is used in the first place. $\endgroup$ Commented Apr 11, 2017 at 23:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.