Suggestions needed about classifier fusion

Question

I'm working on a classification problem which involves two classifier to observe a single event. I'm providing a high level description of the problem without going into the technical details (the classifiers are computer vision related application).

I need to classify the event among N classes, $[C_1, C_2, C_3... C_N]$.

The first classifier, say Classifier A, gives me the probability of a single class. Let us call it $P_A(C_n)$. This is the probability of the observed event belonging to class 'n'. 'n' will be any one value from 1 to N. It does not provide any information about the chances of this event belonging to any other class.

The second Classifier B, however, gives me a complete set of probability $$[P_B(C_1), P_B(C_2), P_B(C_3).... P_B(C_N)]$$. These are the individual probabilities of the event belonging to any of the N classes.

Moreover, for the classifier B, these probabilities of each class is not just one observation. It checks the probability of the event belonging to any class multiple times. That is, for class n, it will give me a set $${P_1^B(C_n), P_2^B(C_n), P_3^B(C_n)..... P_m^B(C_n)}$$, which is the probability of event n, observed/classified m times. ('m' can be between 1 to 10).

Currently, we are assuming the max of these values as the combined probability of class n (for classifier B), i.e,

$$P_B(C_n) = MAX \{P_1^B(C_n), P_2^B(C_n), P_3^B(C_n)..... P_m^B(C_n)\}$$

---

My objective is to combine the results obtained from classifier A and B to arrive at a better estimate of the ground truth. Since A and B works on two different principles and classifies using different input vectors, it is our belief that combining the two of them will help us arrive at a better estimate of the event than using just one of them.

---

Current approach:

What we are doing now is combining the two classifiers using a simple addition. For classifier A, we are arriving at a value for all the other classes (apart from the class for which probability has been derived) as ${(1 - P_A(C_n)) / (N-1)}$.

Then, we're simple adding the probabilities of A and B, i.e,

$$[P_A(C_1) + P_B(C_1), P_A(C_2) + P_B(C_2), P_A(C_3) + P_B(C_3).... P_A(C_N) + P_B(C_N)]$$

This is being followed by another step to normalize the set after addition.

Can you suggest a better approach for combining these two? Any refinement on the current approach? Please help.

Answer 1

There are several approaches to combine classifiers, and in your case, what you are doing is one of them. Also, it is reasonable to expect an improvement in the combined classifier compared to either individual classifiers if they are sufficiently different and complementary from each other. If you had 3 or more classifiers, you could have used techniques like majority-voting or Borda count. However, since you have only two classifiers, you need to go with a combination rule.

The two simple heuristics, with corresponding assumptions on independence of the classifiers, are sum-rule and the product-rule. See this paper for an overview of when each method performs better. In a nutshell, when your underlying feature space is same/highly correlated, and your classifier errors are independent, an average/sum rule is more suited, and when you are training on two different sets of features, a product rule is preferred. However, given the simplicity, trying both and picking the most suitable one is not bad either.

The other approach generalizes the sum-rule by estimating weights for the classifiers. Using the classifier outputs as features, a second-stage classifier is trained. This approach has been proposed under different names. For instance, stacked generalization is one such idea to train a second classifier using the outputs of the first classifier. Other similar approaches are mixture-of-experts, some ensemble learning methods, query-by-committee etc.