2
$\begingroup$

Suppose I have a multi-class classifier like Naive Bayes, k-Nearest Neighbors, Decision Trees, Random Forest, etc.

The classifier maps a feature vector to (let's say) 3 classes: A, B, or C. My problem is that the accuracy I get from this classifier is drastically different per-class, for example:

A: 93%
B: 68%
C: 27%

I would like to achieve equal or similar per-class accuracy, even if it results in lower average accuracy, like:

A: 53%
B: 51%
C: 56%

At the very least, I would like to boost the accuracy on classes that have below-random accuracy.

Most people working on multi-class problems seem to report average accuracy, not minimum per-class accuracy or sensitivity. This paper uses neural nets to address this problem.

My question: Are there systematic ways of adjusting either (1) the input to the classifier, (2) the parameters of the classifier, or (3) the output of a multi-class classifier in order to balance its per-class accuracy?

Note: I'm working with Python's scikit-learn module.

|cite|improve this question
$\endgroup$
  • $\begingroup$ Just as a warning: not all classifiers are created equal. Techniques that work for one might not work for another. $\endgroup$ – shadowtalker Mar 21 '15 at 18:24
  • $\begingroup$ What classifier are you using? Does it output probabilities or just classes? $\endgroup$ – Ben Kuhn Mar 21 '15 at 20:04
  • $\begingroup$ And what fraction of your training data belongs to each class? $\endgroup$ – Ben Kuhn Mar 21 '15 at 20:06
  • $\begingroup$ @BenKuhn I'm getting the same imbalance on each of the classifier I specified above: Naive Bayes, k-Nearest Neighbors, Decision Trees, Random Forest. There is an equal number of training samples for each class (this is achieved by subsampling). $\endgroup$ – Alan Turing Mar 21 '15 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.