2
$\begingroup$

I have two classifiers, say $A$ and $B$ and I am performing 10$\times$10-fold cross-validation on them and record the number of instances correctly classified from the test set. Unfortunately, the number of instances available is quite small, so I have only around 10 instances in each test set. Therefore, the number of correctly classified instances is between 0 and 10, like this: \begin{array}{rrr} \text{fold} & A & B\\ \hline 1 & 8 & 9\\ 2 & 7 & 7\\ 3 & 10 & 8\\ \vdots & \vdots & \vdots\\ 100 & 9 & 8 \end{array}

I would like to know whether $A$ and $B$ have significantly different performance.

Normally (no pun intended), I would use a paired (Welch's) $t$-test for this kind of thing, but the small test set size seems to violate the assumption of normality. Additionally, both $A$ and $B$ are quite good: they have around 90% accuracy on my dataset. This means there are a lot of ties, i.e. "concordant" pairs with $A = B \approx 9$.

It seems Wilcoxon's signed rank test is the staple for problems like this. However, I am concerned with the large number of ties. I have read On Neutral Responses (Zeros) in the Sign Test and Ties in the Wilcoxon-Mann-Whitney Test by Ronald H. Randles recommended in an answer to another question, but I don't really understand how should the parameter $\gamma$ be selected for my problem. Moreover, because Randles only gives a formulation for the rank-sum test, I would need to forget that my observations are paired.

If I were to run leave-one-out cross-validation instead of 10$\times$10-fold, all my responses would be $\{0, 1\}$. In this case, the appropriate test would be McNemars's (exact) test, which, as far as I know, is simply a permuation test. Would preparing a permuation test be appropriate for my data in $[0, 10] \cap \mathbb{Z}$? Because the set of absolute differences is quite small, I could even do an exact permuation test. But is that meaningful?

$\endgroup$
2
$\begingroup$

This problem calls for using a permutation test, a pure non-parametric test that does not have any normality assumptions. Check the "Hot Wings" example in Section 16 of these notes: http://arxiv.org/abs/1603.04929, it is essentially identical to your problem. Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.