I have two classifiers, say $A$ and $B$ and I am performing 10$\times$10-fold cross-validation on them and record the number of instances correctly classified from the test set. Unfortunately, the number of instances available is quite small, so I have only around 10 instances in each test set. Therefore, the number of correctly classified instances is between 0 and 10, like this: \begin{array}{rrr} \text{fold} & A & B\\ \hline 1 & 8 & 9\\ 2 & 7 & 7\\ 3 & 10 & 8\\ \vdots & \vdots & \vdots\\ 100 & 9 & 8 \end{array}
I would like to know whether $A$ and $B$ have significantly different performance.
Normally (no pun intended), I would use a paired (Welch's) $t$-test for this kind of thing, but the small test set size seems to violate the assumption of normality. Additionally, both $A$ and $B$ are quite good: they have around 90% accuracy on my dataset. This means there are a lot of ties, i.e. "concordant" pairs with $A = B \approx 9$.
It seems Wilcoxon's signed rank test is the staple for problems like this. However, I am concerned with the large number of ties. I have read On Neutral Responses (Zeros) in the Sign Test and Ties in the Wilcoxon-Mann-Whitney Test by Ronald H. Randles recommended in an answer to another question, but I don't really understand how should the parameter $\gamma$ be selected for my problem. Moreover, because Randles only gives a formulation for the rank-sum test, I would need to forget that my observations are paired.
If I were to run leave-one-out cross-validation instead of 10$\times$10-fold, all my responses would be $\{0, 1\}$. In this case, the appropriate test would be McNemars's (exact) test, which, as far as I know, is simply a permuation test. Would preparing a permuation test be appropriate for my data in $[0, 10] \cap \mathbb{Z}$? Because the set of absolute differences is quite small, I could even do an exact permuation test. But is that meaningful?
