I will edit this question for more specific questions.
I want to predict the presence of multiple, binary questionnaire items with a classifier, such as SVM. I do this via 10 fold nested cross validation, optimizing hyperparameters in the validation set and evaluating performance on the hold out test set. All CVs are based on the same dataset, thus data is depended which violates an important assumption of standard significance tests. Target Values are available for all subjects, but some targets are highly disbalanced, whereas others are approx. balanced. I chose "balanced accuracy" as the metric the model training optimizes for. I have around 1500 subjects and 2000 Features.
I hypothesize that classification will work better for one group of items (say Item 1-5) than for another group of items (say Item 6-10).
- Is there an appropriate way of testing statistical significance of the difference in the classification results between the two item groups, so that I could conclude something like "classification works significantly better for Items 1-5 as opposed to Items 6-10"? All approaches I looked up so far were about comparisons of single classifiers or multiple classifiers, but not about group differences in classification results.
I thought about McNemar test, corrected paired t test (Nadeau & Bengio, 2003) or non-parametric test like Wilcinson sign-rank test (though the independence assumption is violated).
Using McNemar test for comparing classifiers, it uses a contingency table of the form [[c1,c2 correct, c1 correct c2 incorrect, c1 incorrect c2 correct, c1,c2 inccorect]]. When I now compare two groups of Items (or just two Items, for a simple example), it is possible that the items are differently balanced. Thus, it is possible that the item with better balanced accuracy has in total more misclassifications than the item with a worse balanced accuracy, caused by the higher disbalance of the second item. This is why I think that McNemar test is not really applicable in my case.
As an alternative, I suggest the corrected paired t test, as for example described here: https://medium.com/analytics-vidhya/using-the-corrected-paired-students-t-test-for-comparing-the-performance-of-machine-learning-dc6529eaa97f I tried to adjust this procedure for my task by the following procedure: First, provide a performance estimate for every outer fold per item group by calculating the mean of the performance of the single items per group. Then I calculated the 10 differences in the outer folds performances over the two groups. Finally, I calculated the t and p value with the corrected variance as described by Nadeau & Bengio. This procedure leads to plausible results, variance estimates are very low though.
My new questions are:
Is there a way McNemar test is applicable to targets which are differently balanced with a metric like e. g. balanced accuracy?
Is there an established way of correcting for violation of the independence assumption, e. g. by correcting variance estimates or something like this?
Is it reasonable to assess the group differences I described with a corrected paired t test? I'd also be happy about some feedback about my application of corrected paired t test.
Kind regards :)