test of significance for two groups of classification results

Question

I will edit this question for more specific questions.

I want to predict the presence of multiple, binary questionnaire items with a classifier, such as SVM. I do this via 10 fold nested cross validation, optimizing hyperparameters in the validation set and evaluating performance on the hold out test set. All CVs are based on the same dataset, thus data is depended which violates an important assumption of standard significance tests. Target Values are available for all subjects, but some targets are highly disbalanced, whereas others are approx. balanced. I chose "balanced accuracy" as the metric the model training optimizes for. I have around 1500 subjects and 2000 Features.

I hypothesize that classification will work better for one group of items (say Item 1-5) than for another group of items (say Item 6-10).

Old question:

1. Is there an appropriate way of testing statistical significance of the difference in the classification results between the two item groups, so that I could conclude something like "classification works significantly better for Items 1-5 as opposed to Items 6-10"? All approaches I looked up so far were about comparisons of single classifiers or multiple classifiers, but not about group differences in classification results.

I thought about McNemar test, corrected paired t test (Nadeau & Bengio, 2003) or non-parametric test like Wilcinson sign-rank test (though the independence assumption is violated).

Using McNemar test for comparing classifiers, it uses a contingency table of the form [[c1,c2 correct, c1 correct c2 incorrect, c1 incorrect c2 correct, c1,c2 inccorect]]. When I now compare two groups of Items (or just two Items, for a simple example), it is possible that the items are differently balanced. Thus, it is possible that the item with better balanced accuracy has in total more misclassifications than the item with a worse balanced accuracy, caused by the higher disbalance of the second item. This is why I think that McNemar test is not really applicable in my case.

As an alternative, I suggest the corrected paired t test, as for example described here: https://medium.com/analytics-vidhya/using-the-corrected-paired-students-t-test-for-comparing-the-performance-of-machine-learning-dc6529eaa97f I tried to adjust this procedure for my task by the following procedure: First, provide a performance estimate for every outer fold per item group by calculating the mean of the performance of the single items per group. Then I calculated the 10 differences in the outer folds performances over the two groups. Finally, I calculated the t and p value with the corrected variance as described by Nadeau & Bengio. This procedure leads to plausible results, variance estimates are very low though.

My new questions are:

1. Is there a way McNemar test is applicable to targets which are differently balanced with a metric like e. g. balanced accuracy?

2. Is there an established way of correcting for violation of the independence assumption, e. g. by correcting variance estimates or something like this?

3. Is it reasonable to assess the group differences I described with a corrected paired t test? I'd also be happy about some feedback about my application of corrected paired t test.

Kind regards :)

Answer 1

Yes there is a way to do this. The output between item 1-5 and item 6-10 (assuming they're 2 groups) will yield a 2x2 table of 'classification1' and 'classification2', followed by 0 or 1. The problem then becomes very similar to this question here: Testing binary data

Likely since machine learning is involved I assume the data set will be pretty big (N>1000) so significance testing here would lose a lot of its meaning because even minimal differences are significant. Therefore I suggest looking closer to statistical power such as Phi φ , which would be done with the Chi Square. So if there is a difference in significance you can know how large this difference is.

Answer 2

You should first pick a metric to evaluate the quality of a classification, making the statement "a classification will work better" verifiable. I often choose the F1-score, but that depends on your requirements.

This gives you two distributions over F1-scores, one for the items 1-5 and another for the items 6-10. You now want to know whether those two distributions are significantly different. There are several tests you can choose from, depending on the properties of the distributions. E.g. you could try the two-sample t-test, provided all the preconditions like normality are fulfilled. Also, a rather general test would be Kolmogorov-Smirnov.

Those tests will return p-values. If the p-value is smaller than 0.05, the test is considered significant. Of course, you can be more strict and take a lower threshold, say 0.01. If the p-value is not smaller than your threshold, you cannot conclude anything. In particular, you also cannot conclude that there is no difference.

Note that the size of the p-value doesn't tell you much about the actual size of the difference (effect size), i.e. a smaller p-value does not mean there is a larger difference.