# How to do a significance test on the performance of four classifiers on two different test sets?

I'm trying to find out if an authorship obfuscation technique leads to a statistically significant drop in authorship classification performance.

To test this assumption, I have a dataset of texts split into a training set ($$t_{train}$$) and a test set ($$t_{test}$$). I then produce an obfuscated version of $$t_{test}$$. Let's call that obfuscated version $$t_{obf}$$.

I setup four different classifiers (each trained on $$t_{train}$$) and use them to predict the authors of:

1. The non-obfuscated test set $$t_{test}$$.
2. The obfuscated test set $$t_{obf}$$.

This gives me a 4x2 matrix of (weighted) results. I'm using Accuracy here for illustration purposes but this could also be any other metric (e.g. F1).

Classifier Accuracy on $$t_{test}$$ Accuracy on $$t_{obf}$$
A 0.7 0.2
B 0.5 0.3
C 0.4 0.3
D 0.7 0.5

What is a good way to test if the observed drop in classification performance on the obfuscated test set has statistical significance? I was thinking of a Wilcoxon Signed Rank Test but from what I understand I would need more result pairs (i.e. more classifiers) for that.

To be clear, I don't want to compare the classifiers among each other. I want to test if the obfuscation has a statistically significant impact on the classification performance across these classifiers.

This is more easily done using your raw data: the number of correct and incorrect classifications, for each version of the test set, for each classifier.

A simple approach would be to run four chi-squared tests on the contingency tables for each classifier, e.g.

|            | Test | Obf |
| N Correct  | a    | b   |
| N Incorrect| c    | d   |


which would give you a p-value for each classifier.

A more complete approach would be to use multilevel logistic regression, with estimates allowed to vary between classifiers are random effects, e.g., (using the lme4 package for R):

model = glmer(accuracy ~ 1 + set + (1 + set | classifier),
data = your_data, family = binomial)


where your data has one row per classification case, and set is either 'test' or 'obf'`.

• Thanks for your answer @Eoin! That helps a lot. If I do the four chi-squared tests and end up with four p-values—one per classifier—how would you recommend I best report the results? Is it advisable to do something like Fisher's method to combine the p-values and then only report that combined p-value? Commented Feb 14, 2022 at 22:15
• If you want to combine the results, use the second approach. This will give you estimates and standard errors for each classifier, as well as an overall estimate, SE, and p-value.
– Eoin
Commented Feb 15, 2022 at 9:30
• Thanks @Eoin! Assuming I go with the chi-squared test, what are the requirements for my raw data? Does it need to follow a certain distribution? Commented Apr 23, 2022 at 16:04