CORE QUESTION: I have a single trained classifier (single participant) tested on 2 related multiclass classification tasks. As each trial of the classification tasks are related, the 2 sets of predictions constitute paired data. I would like to run a paired permutation test to find out if the difference in classification accuracy between the 2 prediction sets is significant.
So my data consists of 2 lists of predicted classes, where each prediction is related to the prediction in the other test set at the same index.
actual_classes = [1, 3, 6, 1, 22, 1, 11, 12, 9, 2] predictions1 = [1, 3, 6, 1, 22, 1, 11, 12, 9 10] # 90% acc. predictions2 = [1, 3, 7, 10, 22, 1, 7, 12, 2, 10] # 50% acc.
H0: There is no significant difference in classification accuracy.
ADDITIONAL INFORMATION: In my case, the classifier is a deep neural network model, trained to classify audiovisual videos (but if you're uncomfortable with that, imagine it is a single participant). The model is tested on a held-out test set of audiovisual videos, and then tested on this same video set again, but without any audio data (hence paired). I would like to know if there is a significant difference in classification accuracy.
I am not interested here in cross-validation. I am using a particular held-out test set in a different domain to the training set (and so the entire training set is used for training and the entire test set is used for testing).
I'm comfortable with running a paired permutation test with continuous data (finding the difference between each value in a pair, randomising the sign and then taking the mean). But I'm unsure of how to deal with classification data as I am unsure how to measure the single paired value.
I would really like a paired permutation test here, but failing that, upvotes for anyone providing other tests and advice. Any help appreciated!