1
$\begingroup$

I'm using scikit-learn for a small sample (36) classification problem with three features and three outputs (one output is binary and the other two are ternary).

I'm using separate classifiers for each output, and I'm achieving reasonable classification rates for two outputs, but the remaining one, which is a 3-class problem, is abysmal.

I've tried many different classifiers (SVM, random forest, logistic regression, naive bayes), and using leave-one-out cross-validation the classification rate fluctuates between 0% and 33% (this only happens when the classifier always outputs the same class).

What I don't understand is how it's possible for the classifier to achieve a lower rate then expected by chance. A thought experiment says that when it achieves 0% classification rate, I could ignore the predicted class and randomly choose between the other two, achieving 50% classification rate, which would be better than chance.

Why is this not happening automatically?

$\endgroup$
  • $\begingroup$ Is the data quite biased? And maybe the classifiers need to be tuned. Could you give more information? $\endgroup$ – Vincent Oct 2 '13 at 16:48
  • $\begingroup$ @Vincent The classes are equally divided. For output 1 and 2 I used a linear SVM (C=1.0) and achieved ~63% and ~80% classification rate respectively. Using the same classifier for output 3 I only get a 8.3% rate. (I also used scikit-learn "scale" on the features). Data can be found here: docs.google.com/file/d/0BzpEPYc7dTD0QjN4X3NWRVRWSUk/… $\endgroup$ – Pedro Tabacof Oct 2 '13 at 18:54
3
$\begingroup$

Check whether your leave-one-out procedure leaves out a single sample each time or a triplet of samples (one for each class). If it leaves only a single sample, then your training set is always biased against the test sample. That can explain below chance performance.

For example, consider a classifier that instead of looking at the data, simply outputs the most common label of the training set - that would achieve 0% classification rate in a leave one-sample out scheme. You are of course not using such classifier, but classifiers like SVM do have some susceptibility to bias.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.