# How are the common ways to combine classification results from different classifiers?

I am building a method to classify some data.

There are 4 classes (A,B,D,F). I can achieve only 48% of correct classifications if I use bagged tree. I have tried many other classifiers(e.g. SVM, ensemble KNN, neural network, etc), but bagged tree is the best.

For this classification, there is a fun point. 'A' needs to be classified correctly as much as possible. B,D & F are not important. If the result is 'A', but the prediction is not 'A'. It would be 'NOT good'. The worst case (very bad) is that if it is NOT 'A' but the predicted result is A.

I want to combine the results form a few different classifiers (fusion) to boost the overall result. Any suggestion of how to do that? If you have any other suggestion to do a better prediction, please advise too.

First, we need to incorporate the impact of getting a wrong classification by using a loss function $L(\hat{y}_i,y)$, where $\hat y_i$ is the predicted classification and $y_i$ is the actual classification. In your case, the worst potential loss will be for $L(\hat y_i, \mathrm{A})$ if $\hat y_i \neq \mathrm{A}$
One approach is bagging: Pick some aggregation approach (consensus or more complicated approach). Examples of more a complicated aggregation approach could be one where you set a "bias fraction" $f$ for A, so that the consensus for a "non-A" class must be greater than $f$ (maybe require 60% supermajority not just 51%) or it must be $1+f$ times the conensus fraction for A.