I will edit this question for more specific questions.

I want to predict the presence of multiple, binary questionnaire items with a classifier, such as SVM. I do this via 10 fold nested cross validation, optimizing hyperparameters in the validation set and evaluating performance on the hold out test set. All CVs are based on the same dataset, thus data is depended which violates an important assumption of standard significance tests. Target Values are available for all subjects, but some targets are highly disbalanced, whereas others are approx. balanced. I chose "balanced accuracy" as the metric the model training optimizes for. I have around 1500 subjects and 2000 Features.

I hypothesize that classification will work better for one group of items (say Item 1-5) than for another group of items (say Item 6-10).

Old question:

  1. Is there an appropriate way of testing statistical significance of the difference in the classification results between the two item groups, so that I could conclude something like "classification works significantly better for Items 1-5 as opposed to Items 6-10"? All approaches I looked up so far were about comparisons of single classifiers or multiple classifiers, but not about group differences in classification results.

I thought about McNemar test, corrected paired t test (Nadeau & Bengio, 2003) or non-parametric test like Wilcinson sign-rank test (though the independence assumption is violated).

Using McNemar test for comparing classifiers, it uses a contingency table of the form [[c1,c2 correct, c1 correct c2 incorrect, c1 incorrect c2 correct, c1,c2 inccorect]]. When I now compare two groups of Items (or just two Items, for a simple example), it is possible that the items are differently balanced. Thus, it is possible that the item with better balanced accuracy has in total more misclassifications than the item with a worse balanced accuracy, caused by the higher disbalance of the second item. This is why I think that McNemar test is not really applicable in my case.

As an alternative, I suggest the corrected paired t test, as for example described here: https://medium.com/analytics-vidhya/using-the-corrected-paired-students-t-test-for-comparing-the-performance-of-machine-learning-dc6529eaa97f I tried to adjust this procedure for my task by the following procedure: First, provide a performance estimate for every outer fold per item group by calculating the mean of the performance of the single items per group. Then I calculated the 10 differences in the outer folds performances over the two groups. Finally, I calculated the t and p value with the corrected variance as described by Nadeau & Bengio. This procedure leads to plausible results, variance estimates are very low though.

My new questions are:

  1. Is there a way McNemar test is applicable to targets which are differently balanced with a metric like e. g. balanced accuracy?

  2. Is there an established way of correcting for violation of the independence assumption, e. g. by correcting variance estimates or something like this?

  3. Is it reasonable to assess the group differences I described with a corrected paired t test? I'd also be happy about some feedback about my application of corrected paired t test.

Kind regards :)


2 Answers 2


Yes there is a way to do this. The output between item 1-5 and item 6-10 (assuming they're 2 groups) will yield a 2x2 table of 'classification1' and 'classification2', followed by 0 or 1. The problem then becomes very similar to this question here: Testing binary data

Likely since machine learning is involved I assume the data set will be pretty big (N>1000) so significance testing here would lose a lot of its meaning because even minimal differences are significant. Therefore I suggest looking closer to statistical power such as Phi φ , which would be done with the Chi Square. So if there is a difference in significance you can know how large this difference is.

  • $\begingroup$ As I mentioned also in my comment regarding the answer of @frank, such a test will involve an independence assumption that is most likely violated here, so I think it will not be valid. $\endgroup$ Feb 8, 2022 at 11:16
  • $\begingroup$ You're right, in that case you can use McNemar's test and use odds ratio to test for how big the difference is. $\endgroup$ Feb 8, 2022 at 13:38
  • $\begingroup$ Thank you for the answer! I am aware of the limitations of signifiacance testing. I've thought about using McNemar test, but I wasn't sure if it is appropriate to just sum over the confusion matrices of the two groups. $\endgroup$ Feb 8, 2022 at 13:59
  • $\begingroup$ @cinderashes Just a further inquiry. When I use the McNemar test as e. g. described here: machinelearningmastery.com/mcnemars-test-for-machine-learning/… I need a contingency table of the form [[c1,c2 correct, c1 correct c2 incorrect, c1 incorrect c2 correct, c1,c2 inccorect]] How do I get there from single item predictions? Just summing over items and than dichotomizing the results? $\endgroup$ Feb 10, 2022 at 16:10

You should first pick a metric to evaluate the quality of a classification, making the statement "a classification will work better" verifiable. I often choose the F1-score, but that depends on your requirements.

This gives you two distributions over F1-scores, one for the items 1-5 and another for the items 6-10. You now want to know whether those two distributions are significantly different. There are several tests you can choose from, depending on the properties of the distributions. E.g. you could try the two-sample t-test, provided all the preconditions like normality are fulfilled. Also, a rather general test would be Kolmogorov-Smirnov.

Those tests will return p-values. If the p-value is smaller than 0.05, the test is considered significant. Of course, you can be more strict and take a lower threshold, say 0.01. If the p-value is not smaller than your threshold, you cannot conclude anything. In particular, you also cannot conclude that there is no difference.

Note that the size of the p-value doesn't tell you much about the actual size of the difference (effect size), i.e. a smaller p-value does not mean there is a larger difference.

  • $\begingroup$ Chances are items 1-5 won't be independent of items 6-10. In that case the chosen t-test should be paired and Kolmogorov-Smirnov will not apply. Note also that even when using a paired t-test the individual pairs of F1-entries need to be independent, which chances are you won't have using CV-folds, only when running things in an isolated manner on independent sub-samples. $\endgroup$ Feb 8, 2022 at 11:13
  • $\begingroup$ Thank you for the answer! I am aware of limitations of p-values. I chose balanced accuracy as a metric due to some imbalanced classes. Independence is an issue, as Christian mentioned and I think assumptions of a paired t test probably won't hold, so I would stick to the McNemar test, as suggested by cinderashes $\endgroup$ Feb 8, 2022 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.