How are classifications merged in an ensemble classifier?

How does an ensemble classifier merge the predictions of its constituent classifiers? I'm having difficulty finding a clear description. In some code examples I've found, the ensemble just averages the predictions, but I don't see how this could possible make a "better" overall accuracy.

Consider the following case. An ensemble classifier is composed of 10 classifiers. One classifier is has an accuracy of 100% of the time in data subset X, and 0% all other times. All other classifiers have an accuracy of 0% in data subset X, and 100% all other times.

Using an averaging formula, where classifier accuracy is ignored, the ensemble classifier would have, at best, 50% accuracy. Is this correct, or am I missing something? How can taking the average prediction from N potentially clueless classifiers possibly create a better prediction then a single classifier that's an expert in a specific domain?

You are missing the fact that "bad" classifier do not have 0% accuracy, rather it is not significantly better than a random guessing.

This way good predictions are always the same and accumulate (since the truth is only one) while bad predictions are random noise which average out.

• It's not that any classifier is overall "bad" or "good". I'm talking about each classifier's accuracy in specific domains. If one classifier is perfect in a specific domain, then including it in an ensemble may potentially obscure it's usefulness, because all the other classifiers may make bad classifications. – Cerin Jan 22 '12 at 21:21
• Unless you have perfect contradictions (which are almost absent in real, usable data), domains do not change anything since the classifier should also produce noise instead of exclusively bad class for objects outside its domain -- this way only different partial classifiers will be "good" in each of the domains. Anyway, it is obvious that a sharp, perfect signal will be silenced by the noise (hopefully not completely), but this is the trade-off for having no information about what partial classifiers work for what objects. – mbq Jan 22 '12 at 23:51
• My point is that if you take a great classifier and average it's prediction with bad classifiers, you're unlikely to get a better prediction. You're diluting your good prediction. – Cerin Jan 31 '12 at 17:31
• @chriss - usually when you average you weight each classifier according to its performance. in this scheme a great classifier only gets diluted by other great classifiers. Crap classifiers get low weight (usually exponentially low) and so dont dilute. – probabilityislogic Feb 8 '12 at 3:18
• @ChisS You are perfectly right -- the point of ensemble is that you can't cherry-pick best classifier for each object, so you average trading reasonable solution for some signal attenuation by noise and the risk of correlated-base-classifiers catastrophe. – mbq Feb 8 '12 at 10:29

I read a clear example from Introduction to Data Mining by Tan et al.

The example claims that if you are combining your classifiers with a voting system, that is classify a record with the most voted class, you obtain better performance. However, this example uses directly the output label of classifiers, and not the predictions (I think you meant probabilities).

Let's have 25 independent classifiers that have generalization error $e = 1 - \mbox{accuracy} = 0.35$. In order to misclassify a record at least half of them have to misclassify it.

Everything can be modeled with random variables, but you just have to compute the probability that at least 13 of them misclassify the record $$\sum_{i=13}^{25}\binom{25}{i}e^i(1-e)^{(25-i)} = 0.06$$ where each term of the summation means that $i$ classifier get the record class correctly and $25-i$ get it wrong.

Using directly predictions and using as a combination method an average, I think that it could be a bit more difficult to show the improvment in ensemble performance. However, focusing only on predictions and without caring at the output label of the ensemble, averaging more predictions can be seen as an estimator of the real probability. Therefore, adding classifiers should improve the predictions of the ensemble technique.

• This is a great way to understand why the ensemble works. However the specific case is likely to be too optimistic in terms of improved performance. This is basic each classifier is trained (usually) on the same data - making the independence of the classifiers questionable. – probabilityislogic Feb 8 '12 at 3:14
• Of course, the independence is a too strong hypothesis. – Simone Feb 8 '12 at 9:52

In case of classification generally there are two ways to ensemble the prediction. Lets say it's a binary class classification problem and you have 3 models to ensemble called m1,m2 and m3 and the training dataset is called train and testing dataset called test.Models are already build on train.Then a python code will be as following.

First method is to take a round of the average

pred=round([m1.predict(test)+m2.predict(test)+m3.predict(test)]/3)


So the output will be a vector of value 0 and 1

Second method is to ensemble the prediction probability of each class from these models and ensemble that and then decide the class either on the basis of a hard threshold or some logic.

pred_proba=[m1.predict(test).predict_proba++ m2.predict(test).predict_proba
+m3.predict(test).predict_proba]/3
# Simple average ensemble,however you can try weighted average as well


iterate through the entire pred_proba vector to find which one in 0 and which 1 is 1 basing on hard threshold 0.5

pred=[] # Initialize a blank list for prediction

for x in pred_proba:
if x>0.5:
pred.append[1]
else:
pred.append[0]


So pred is the final ensemble prediction.