I'm working on a classification problem which involves two classifier to observe a single event. I'm providing a high level description of the problem without going into the technical details (the classifiers are computer vision related application).

I need to classify the event among N classes, $[C_1, C_2, C_3... C_N]$.

The first classifier, say Classifier A, gives me the probability of a single class. Let us call it $P_A(C_n)$. This is the probability of the observed event belonging to class 'n'. 'n' will be any one value from 1 to N. It does not provide any information about the chances of this event belonging to any other class.

The second Classifier B, however, gives me a complete set of probability $$[P_B(C_1), P_B(C_2), P_B(C_3).... P_B(C_N)]$$. These are the individual probabilities of the event belonging to any of the N classes.

Moreover, for the classifier B, these probabilities of each class is not just one observation. It checks the probability of the event belonging to any class multiple times. That is, for class n, it will give me a set $${P_1^B(C_n), P_2^B(C_n), P_3^B(C_n)..... P_m^B(C_n)}$$, which is the probability of event n, observed/classified m times. ('m' can be between 1 to 10).

Currently, we are assuming the max of these values as the combined probability of class n (for classifier B), i.e,

$$P_B(C_n) = MAX \{P_1^B(C_n), P_2^B(C_n), P_3^B(C_n)..... P_m^B(C_n)\}$$

My objective is to combine the results obtained from classifier A and B to arrive at a better estimate of the ground truth. Since A and B works on two different principles and classifies using different input vectors, it is our belief that combining the two of them will help us arrive at a better estimate of the event than using just one of them.

Current approach:

What we are doing now is combining the two classifiers using a simple addition. For classifier A, we are arriving at a value for all the other classes (apart from the class for which probability has been derived) as ${(1 - P_A(C_n)) / (N-1)}$.

Then, we're simple adding the probabilities of A and B, i.e,

$$[P_A(C_1) + P_B(C_1), P_A(C_2) + P_B(C_2), P_A(C_3) + P_B(C_3).... P_A(C_N) + P_B(C_N)]$$

This is being followed by another step to normalize the set after addition.

Can you suggest a better approach for combining these two? Any refinement on the current approach? Please help.


1 Answer 1


There are several approaches to combine classifiers, and in your case, what you are doing is one of them. Also, it is reasonable to expect an improvement in the combined classifier compared to either individual classifiers if they are sufficiently different and complementary from each other. If you had 3 or more classifiers, you could have used techniques like majority-voting or Borda count. However, since you have only two classifiers, you need to go with a combination rule.

The two simple heuristics, with corresponding assumptions on independence of the classifiers, are sum-rule and the product-rule. See this paper for an overview of when each method performs better. In a nutshell, when your underlying feature space is same/highly correlated, and your classifier errors are independent, an average/sum rule is more suited, and when you are training on two different sets of features, a product rule is preferred. However, given the simplicity, trying both and picking the most suitable one is not bad either.

The other approach generalizes the sum-rule by estimating weights for the classifiers. Using the classifier outputs as features, a second-stage classifier is trained. This approach has been proposed under different names. For instance, stacked generalization is one such idea to train a second classifier using the outputs of the first classifier. Other similar approaches are mixture-of-experts, some ensemble learning methods, query-by-committee etc.

  • $\begingroup$ Using this paper:darwin.cwru.edu/ref/view.php?id=316&article=Elston+Reprints, we used Fischer's method of combining two classifiers, i.e. (-cln c + c), where c = Pa*Pb, and found that this method creates an output which is very similar to simple product-rule. In fact, sum-rule does produce better result than product-rule or Fisher's rule in some cases. Our feature space should be highly correlated (although we haven't checked it for correlation, this is just an estimate), but we also want the fusion classifier to be resilient to bogus results from any one of the classifiers. $\endgroup$
    – metsburg
    Sep 24, 2014 at 5:22
  • $\begingroup$ Do you think Borda count would be a good approach for combination? I'm also considering deciding the classifier fusion technique on the run time, perhaps we can take the feature set from both the classifiers and check for Pearson coefficient or some other factor (to identify the degree of correlation) and then branch to the suitable fuser (Sum-rule/Fischer's/Borda count) based on this. What do you think? $\endgroup$
    – metsburg
    Sep 24, 2014 at 5:24
  • $\begingroup$ Stacked generalization also sounds like a nice idea, since we have some feature vectors which are not used by any of the classifiers. We were wondering how to use the information that has been left-out. A stacked classifier might help us deal with it. $\endgroup$
    – metsburg
    Sep 24, 2014 at 5:43
  • $\begingroup$ Stacked generalization is better in my opinion too, since it is data-driven and not rule-based. Also, it is hard to make a general statement about which combination rule works. My approach would be to vet them based on their empirical performance. $\endgroup$ Sep 24, 2014 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.