I am working on a 3-class classification problem. The classifier I'm using is Bayesian Networks which provides me with a classification accuracy of around 60%. When I do a two-class classification, I get 80% accuracy for differentiating between class 0 & class 1 and between class 0 & class 2. Also, I get only 60% accuracy for classification between class 1 & class 2. I believe the best way to do 3-class classification in this case would be combining the 2 two-class classifiers with 80% accuracy. What comes to my mind is using some sort weighted averaging scheme on the results of the two individual two-class classifiers. I have not solved such a problem in the past and am facing a dilemma as to how I should implement this. Any help/suggestions in this regard would be highly appreciated. Please fell free to suggest other alternatives if you think they may work.
4 Answers
I've done something like this using either of the following:
- (a) Given three different classes (e.g. A, B, C), create an input column for each class. Place '1' in the A column if the sample is an A, '0' otherwise - do this for B and C classes using the same logic. The foregoing columns will be your target fields for three separate binary classifiers (a classifier for A, B, and C).
(b) Feed the predictions - in addition to any other features - into a third classifier, a multiclass classifier whose target is the tri-level target.
- Taking the same approach as 1(a), take the predictions and use rule-based logic (or misclassification costs) to separate the class predictions - this is to avoid ending up with the same sample being predicted as both A and B, both A and C, etc.
Two classifiers which do 0 vs 1 and 0 vs 2 classifications intuitively should perform better than a classifier which has to distinguish between all three at once. The intuition being, that the choice of which 2-classifier to use for a given sample is also to be learned when doing the 0 vs 1 vs 2 classification problem.
I nice paper I found which might help was Fitted Learning: Models with Awareness of their Limits.
It takes a simple neural network, the Feed forward kind but the key idea is that instead of teaching it to predict a vector [0,0,1], [0, 1, 0] or [1, 0, 0] you teach it to predict another vector.
You choose an arbitrary number (say 2) and then the targets you need to predict for each class follow a simple mapping.
[0, 0, 1] -> [0, 0, 0.5, 0, 0, 0.5]
[0, 1, 0] -> [0, 0.5, 0, 0, 0.5, 0]
[1, 0, 0] -> [0.5, 0, 0, 0.5, 0, 0]
That allows you to learn a much cleaner classification. I'd recommend going through the paper and seeing if it helps your problem.
First of all, regarding terminology, you are taking about using multiple two-class classifiers, rather than one-class classifiers. One class classifiers are a class of models used for anomaly or novelty detection, where you have data coming only from a single class. If you have two classes, it's a two-class classifier.
What you want to do is one-vs-rest strategy in multiclass classification. Scikit-learn has nice documentation on such classifiers and API that allows for fitting such classifiers out of the box. You don't need to do weighted averaging, each of the individual classifiers will return some kind of score (often a probability) so for each class you would have the score for this class vs other classes, to make a prediction, you pick the class that has the highest score.
I am not so familiar with Bayes Networks. If you are interested in learning a weighting scheme, I'd propose a meta-linear model to combine those outputs.
A perceptron or linear support vector machine may work well here.