How are classifications merged in an ensemble classifier?

Question

How does an ensemble classifier merge the predictions of its constituent classifiers? I'm having difficulty finding a clear description. In some code examples I've found, the ensemble just averages the predictions, but I don't see how this could possible make a "better" overall accuracy.

Consider the following case. An ensemble classifier is composed of 10 classifiers. One classifier is has an accuracy of 100% of the time in data subset X, and 0% all other times. All other classifiers have an accuracy of 0% in data subset X, and 100% all other times.

Using an averaging formula, where classifier accuracy is ignored, the ensemble classifier would have, at best, 50% accuracy. Is this correct, or am I missing something? How can taking the average prediction from N potentially clueless classifiers possibly create a better prediction then a single classifier that's an expert in a specific domain?

Answer 1

I read a clear example from Introduction to Data Mining by Tan et al.

The example claims that if you are combining your classifiers with a voting system, that is classify a record with the most voted class, you obtain better performance. However, this example uses directly the output label of classifiers, and not the predictions (I think you meant probabilities).

Let's have 25 independent classifiers that have generalization error $e = 1 - \mbox{accuracy} = 0.35$. In order to misclassify a record at least half of them have to misclassify it.

Everything can be modeled with random variables, but you just have to compute the probability that at least 13 of them misclassify the record $$\sum_{i=13}^{25}\binom{25}{i}e^i(1-e)^{(25-i)} = 0.06$$ where each term of the summation means that $i$ classifier get the record class correctly and $25-i$ get it wrong.

Using directly predictions and using as a combination method an average, I think that it could be a bit more difficult to show the improvment in ensemble performance. However, focusing only on predictions and without caring at the output label of the ensemble, averaging more predictions can be seen as an estimator of the real probability. Therefore, adding classifiers should improve the predictions of the ensemble technique.

Answer 2

You are missing the fact that "bad" classifier do not have 0% accuracy, rather it is not significantly better than a random guessing.

This way good predictions are always the same and accumulate (since the truth is only one) while bad predictions are random noise which average out.

Answer 3

In case of classification generally there are two ways to ensemble the prediction. Lets say it's a binary class classification problem and you have 3 models to ensemble called m1,m2 and m3 and the training dataset is called train and testing dataset called test.Models are already build on train.Then a python code will be as following.

First method is to take a round of the average

pred=round([m1.predict(test)+m2.predict(test)+m3.predict(test)]/3) 

So the output will be a vector of value 0 and 1

Second method is to ensemble the prediction probability of each class from these models and ensemble that and then decide the class either on the basis of a hard threshold or some logic.

pred_proba=[m1.predict(test).predict_proba++ m2.predict(test).predict_proba 
+m3.predict(test).predict_proba]/3 
# Simple average ensemble,however you can try weighted average as well

iterate through the entire pred_proba vector to find which one in 0 and which 1 is 1 basing on hard threshold 0.5

pred=[] # Initialize a blank list for prediction

for x in pred_proba:
 if x>0.5:
   pred.append[1]
 else:
   pred.append[0]

So pred is the final ensemble prediction.