# How are classifications merged in an ensemble classifier?

How does an ensemble classifier merge the predictions of its constituent classifiers? I'm having difficulty finding a clear description. In some code examples I've found, the ensemble just averages the predictions, but I don't see how this could possible make a "better" overall accuracy.

Consider the following case. An ensemble classifier is composed of 10 classifiers. One classifier is has an accuracy of 100% of the time in data subset X, and 0% all other times. All other classifiers have an accuracy of 0% in data subset X, and 100% all other times.

Using an averaging formula, where classifier accuracy is ignored, the ensemble classifier would have, at best, 50% accuracy. Is this correct, or am I missing something? How can taking the average prediction from N potentially clueless classifiers possibly create a better prediction then a single classifier that's an expert in a specific domain?

-

I read a clear example from Introduction to Data Mining by Tan et al.

The example claims that if you are combining your classifiers with a voting system, that is classify a record with the most voted class, you obtain better performance. However, this example uses directly the output label of classifiers, and not the predictions (I think you meant probabilities).

Let's have 25 independent classifiers that have generalization error $e = 1 - \mbox{accuracy} = 0.35$. In order to misclassify a record at least half of them have to misclassify it.

Everything can be modeled with random variables, but you just have to compute the probability that at least 13 of them misclassify the record $$\sum_{i=13}^{25}\binom{25}{i}e^i(1-e)^{(25-i)} = 0.06$$ where each term of the summation means that $i$ classifier get the record class correctly and $25-i$ get it wrong.

Using directly predictions and using as a combination method an average, I think that it could be a bit more difficult to show the improvment in ensemble performance. However, focusing only on predictions and without caring at the output label of the ensemble, averaging more predictions can be seen as an estimator of the real probability. Therefore, adding classifiers should improve the predictions of the ensemble technique.

-
This is a great way to understand why the ensemble works. However the specific case is likely to be too optimistic in terms of improved performance. This is basic each classifier is trained (usually) on the same data - making the independence of the classifiers questionable. –  probabilityislogic Feb 8 '12 at 3:14
Of course, the independence is a too strong hypothesis. –  Simone Feb 8 '12 at 9:52