0
$\begingroup$

I'm trying to find out if an authorship obfuscation technique leads to a statistically significant drop in authorship classification performance.

To test this assumption, I have a dataset of texts split into a training set ($t_{train}$) and a test set ($t_{test}$). I then produce an obfuscated version of $t_{test}$. Let's call that obfuscated version $t_{obf}$.

I setup four different classifiers (each trained on $t_{train}$) and use them to predict the authors of:

  1. The non-obfuscated test set $t_{test}$.
  2. The obfuscated test set $t_{obf}$.

This gives me a 4x2 matrix of (weighted) results. I'm using Accuracy here for illustration purposes but this could also be any other metric (e.g. F1).

Classifier Accuracy on $t_{test}$ Accuracy on $t_{obf}$
A 0.7 0.2
B 0.5 0.3
C 0.4 0.3
D 0.7 0.5

What is a good way to test if the observed drop in classification performance on the obfuscated test set has statistical significance? I was thinking of a Wilcoxon Signed Rank Test but from what I understand I would need more result pairs (i.e. more classifiers) for that.

To be clear, I don't want to compare the classifiers among each other. I want to test if the obfuscation has a statistically significant impact on the classification performance across these classifiers.

$\endgroup$

1 Answer 1

0
+100
$\begingroup$

This is more easily done using your raw data: the number of correct and incorrect classifications, for each version of the test set, for each classifier.

A simple approach would be to run four chi-squared tests on the contingency tables for each classifier, e.g.

|            | Test | Obf |
| N Correct  | a    | b   |
| N Incorrect| c    | d   |

which would give you a p-value for each classifier.

A more complete approach would be to use multilevel logistic regression, with estimates allowed to vary between classifiers are random effects, e.g., (using the lme4 package for R):

model = glmer(accuracy ~ 1 + set + (1 + set | classifier), 
              data = your_data, family = binomial)`

where your data has one row per classification case, and set is either 'test' or 'obf'.

$\endgroup$
3
  • $\begingroup$ Thanks for your answer @Eoin! That helps a lot. If I do the four chi-squared tests and end up with four p-values—one per classifier—how would you recommend I best report the results? Is it advisable to do something like Fisher's method to combine the p-values and then only report that combined p-value? $\endgroup$
    – dnlggr
    Feb 14 at 22:15
  • $\begingroup$ If you want to combine the results, use the second approach. This will give you estimates and standard errors for each classifier, as well as an overall estimate, SE, and p-value. $\endgroup$
    – Eoin
    Feb 15 at 9:30
  • $\begingroup$ Thanks @Eoin! Assuming I go with the chi-squared test, what are the requirements for my raw data? Does it need to follow a certain distribution? $\endgroup$
    – dnlggr
    Apr 23 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.