I'm using scikit-learn for a small sample (36) classification problem with three features and three outputs (one output is binary and the other two are ternary).
I'm using separate classifiers for each output, and I'm achieving reasonable classification rates for two outputs, but the remaining one, which is a 3-class problem, is abysmal.
I've tried many different classifiers (SVM, random forest, logistic regression, naive bayes), and using leave-one-out cross-validation the classification rate fluctuates between 0% and 33% (this only happens when the classifier always outputs the same class).
What I don't understand is how it's possible for the classifier to achieve a lower rate then expected by chance. A thought experiment says that when it achieves 0% classification rate, I could ignore the predicted class and randomly choose between the other two, achieving 50% classification rate, which would be better than chance.
Why is this not happening automatically?