I am working on a 3-class classification problem. The classifier I'm using is Bayesian Networks which provides me with a classification accuracy of around 60%. When I do a two-class classification, I get 80% accuracy for differentiating between class 0 & class 1 and between class 0 & class 2. Also, I get only 60% accuracy for classification between class 1 & class 2. I believe the best way to do 3-class classification in this case would be combining the 2 two-class classifiers with 80% accuracy. What comes to my mind is using some sort weighted averaging scheme on the results of the two individual two-class classifiers. I have not solved such a problem in the past and am facing a dilemma as to how I should implement this. Any help/suggestions in this regard would be highly appreciated. Please fell free to suggest other alternatives if you think they may work.
I've done something like this using either of the following:
- (a) Given three different classes (e.g. A, B, C), create an input column for each class. Place '1' in the A column if the sample is an A, '0' otherwise - do this for B and C classes using the same logic. The foregoing columns will be your target fields for three separate binary classifiers (a classifier for A, B, and C).
(b) Feed the predictions - in addition to any other features - into a third classifier, a multiclass classifier whose target is the tri-level target.
- Taking the same approach as 1(a), take the predictions and use rule-based logic (or misclassification costs) to separate the class predictions - this is to avoid ending up with the same sample being predicted as both A and B, both A and C, etc.
Two classifiers which do 0 vs 1 and 0 vs 2 classifications intuitively should perform better than a classifier which has to distinguish between all three at once. The intuition being, that the choice of which 2-classifier to use for a given sample is also to be learned when doing the 0 vs 1 vs 2 classification problem.
I nice paper I found which might help was Fitted Learning: Models with Awareness of their Limits.
It takes a simple neural network, the Feed forward kind but the key idea is that instead of teaching it to predict a vector [0,0,1], [0, 1, 0] or [1, 0, 0] you teach it to predict another vector.
You choose an arbitrary number (say 2) and then the targets you need to predict for each class follow a simple mapping.
[0, 0, 1] -> [0, 0, 0.5, 0, 0, 0.5] [0, 1, 0] -> [0, 0.5, 0, 0, 0.5, 0] [1, 0, 0] -> [0.5, 0, 0, 0.5, 0, 0]
That allows you to learn a much cleaner classification. I'd recommend going through the paper and seeing if it helps your problem.