I am working on a 3-class classification problem. The classifier I'm using is Bayesian Networks which provides me with a classification accuracy of around 60%. When I do a two-class classification, I get 80% accuracy for differentiating between class 0 & class 1 and between class 0 & class 2. Also, I get only 60% accuracy for classification between class 1 & class 2. I believe the best way to do 3-class classification in this case would be combining the 2 two-class classifiers with 80% accuracy. What comes to my mind is using some sort weighted averaging scheme on the results of the two individual two-class classifiers. I have not solved such a problem in the past and am facing a dilemma as to how I should implement this. Any help/suggestions in this regard would be highly appreciated. Please fell free to suggest other alternatives if you think they may work.
I am not so familiar with Bayes Networks. If you are interested in learning a weighting scheme, I'd propose a meta-linear model to combine those outputs.
A perceptron or linear support vector machine may work well here.
I've done something like this using either of the following:
- (a) Given three different classes (e.g. A, B, C), create an input column for each class. Place '1' in the A column if the sample is an A, '0' otherwise - do this for B and C classes using the same logic. The foregoing columns will be your target fields for three separate binary classifiers (a classifier for A, B, and C).
(b) Feed the predictions - in addition to any other features - into a third classifier, a multiclass classifier whose target is the tri-level target.
- Taking the same approach as 1(a), take the predictions and use rule-based logic (or misclassification costs) to separate the class predictions - this is to avoid ending up with the same sample being predicted as both A and B, both A and C, etc.
Two classifiers which do 0 vs 1 and 0 vs 2 classifications intuitively should perform better than a classifier which has to distinguish between all three at once. The intuition being, that the choice of which 2-classifier to use for a given sample is also to be learned when doing the 0 vs 1 vs 2 classification problem.
I nice paper I found which might help was Fitted Learning: Models with Awareness of their Limits.
It takes a simple neural network, the Feed forward kind but the key idea is that instead of teaching it to predict a vector [0,0,1], [0, 1, 0] or [1, 0, 0] you teach it to predict another vector.
You choose an arbitrary number (say 2) and then the targets you need to predict for each class follow a simple mapping.
[0, 0, 1] -> [0, 0, 0.5, 0, 0, 0.5] [0, 1, 0] -> [0, 0.5, 0, 0, 0.5, 0] [1, 0, 0] -> [0.5, 0, 0, 0.5, 0, 0]
That allows you to learn a much cleaner classification. I'd recommend going through the paper and seeing if it helps your problem.