0
$\begingroup$

I am building a method to classify some data.

There are 4 classes (A,B,D,F). I can achieve only 48% of correct classifications if I use bagged tree. I have tried many other classifiers(e.g. SVM, ensemble KNN, neural network, etc), but bagged tree is the best.

For this classification, there is a fun point. 'A' needs to be classified correctly as much as possible. B,D & F are not important. If the result is 'A', but the prediction is not 'A'. It would be 'NOT good'. The worst case (very bad) is that if it is NOT 'A' but the predicted result is A.

I want to combine the results form a few different classifiers (fusion) to boost the overall result. Any suggestion of how to do that? If you have any other suggestion to do a better prediction, please advise too.

$\endgroup$
0
$\begingroup$

First, we need to incorporate the impact of getting a wrong classification by using a loss function $L(\hat{y}_i,y)$, where $\hat y_i$ is the predicted classification and $y_i$ is the actual classification. In your case, the worst potential loss will be for $L(\hat y_i, \mathrm{A})$ if $\hat y_i \neq \mathrm{A}$

One approach is bagging: Pick some aggregation approach (consensus or more complicated approach). Examples of more a complicated aggregation approach could be one where you set a "bias fraction" $f$ for A, so that the consensus for a "non-A" class must be greater than $f$ (maybe require 60% supermajority not just 51%) or it must be $1+f$ times the conensus fraction for A.

Split data into K folds, train classifiers on the K-1 training folds, and then test your aggregation methods by feeding test data to trained classifiers and calculating the loss of each prediction. Take the average loss as your metric.

Now, you'll need to adjust your method and see how well it does under cross validation. If you're going to be doing a lot of comparisons, I'd suggest holding out a validation set for testing the final, chosen model (trained on the entire dataset (minus validation points)).

$\endgroup$
  • $\begingroup$ I understand the idea. May I know how I should set in MATLAB to achieve 60% supermajority in classifier? $\endgroup$ – Marco Jun 20 '17 at 4:37
  • $\begingroup$ @Marco I don't work in MATLAB, so can't point you towards something. However, you won't "achieve" 60% accuracy, if that's what you thought I said. I was describing a decision rule whereby you default to "A" unless some other label gets over 60% of the votes. $\endgroup$ – user145807 Jun 20 '17 at 4:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.