I have data from 50 human subjects, who are divided into groups A and B (30 participants are in group A and 20 participants in group B). I also have a range of measurements from each subject. I have used a machine learning algorithm (SVM) to predict the subject groups from the features using leave-one-out cross-validation.
I would like to have a p-value for my classifier. If I have understood correctly, I can treat each prediction as a Bernoulli trial, and test for significance using a binomial test. However, I'm a little confused about the details. The paper (pdf) I was following, in the section 'result significance', assumes that the subject groups are of equal size. That's not the case here. What do I need to do to make the test work?