How to combine predictions from ensemble learning

Question

Suppose I have three models, model-1, model-2, model-3 for binary classification. Suppose model-1 has $a_1$ accuracy, model-2 has $a_2$ accuracy, model-3 has $a_3$ accuracy.

For some test data, model-1 classifies it as $1$, whereas model-2 and model-3 classify it as $0$.

What are the best heuristic ways to combine the result and decide the final classification ?

One of the naive way to do this would be :

We can create map $M$ where $M[1]=1$ and $M[0]=-1$. Let we denote result of model-1 as $r_1$, model-2 as $r_2$, model-3 as $r_3$. Let us define $S = a_1*M[r_1]+a_2*M[r_2]+a_3*M[r_3]$. If $S>0$ we can declare final result as $1$ else $0$.

Problem with this is it won't give accurate result with large number of models. Suppose I have $100$ models,also suppose first model has accuracy of $0.9$ and other $99$ models have accuracy of $0.1$ then final result might get wrong.

One way to resolve this is we can only allow those models whose accuracy is not less than $H-0.1$ where $H$ is highest accuracy of any model.

What are some best way to combine the results from different models ? Are there some libraries in python to do this ? I found some of way to do ensemble learning but I want to keep some weightage proportional to accuracy of models (based on previous performance on test data) and not do just some kind of averaging.

Answer 1

First off, don't use accuracy, which is a seriously misleading evaluation measure. Instead, use probabilistic classifications, and evaluate these using proper scoring rules.

You can easily combine probabilistic classifications. If your three models yield predicted probabilities $\hat{p}_1, \hat{p}_2$ and $\hat{p}_3$ for a new instance to belong to the target class, simply take the average of the $\hat{p}_i$ as the combined probabilistic prediction.

If your models differ on some proper scoring rule, then you can use a weighting based on these scores. Or you could even run a logistic regression of your target on the in-sample probabilistic classifications of your three models.

Note, however, that any attempt to estimate weights for combinations will introduce additional variance, and the end result may well be worse than if you had used a simple unweighted average (e.g., Claeskens et al., 2016).

Answer 2

I upgraded your question due to its more general importance in practice.

As background, I am in the process of filing an algorithmic patent that asserts the falsity of the following statement for many distributions:

Note, however, that any attempt to estimate weights for combinations will introduce additional variance, and the end result may well be worse than if you had used a simple unweighted average.

The apparent secret to combining relates quite intuitively, and applies particularly, to small samples where robust methods can suggest new information content that can apparently increase accuracy.

As a simple example, consider the mode of a distribution (which equals, or is mathematically related to, the mean) for which one wants an improved mean estimate over the sample mean. As such, a clustering of points in the middle range of the sorted sample (where one would expect the mode) can be quite informative in shifting a combined estimator towards the true mean and away from the sample mean which is easily distorted by outliers and in small unrepresented samples. The process is, however, as expected heavily dependent on the parent distribution and sample size.

My comment relates to my analysis that was confirmed over repeated simulations for over a dozen distributions commonly found in applications. Quite simply, it is a question of new robust information content introduced. Adding it improves, and surprisingly, it is not a major issue on how to precisely weigh it, as in the effort of discovering/inventing the new information metric.

As to your question, my suggested answer for large data sets that appear relevant to your decision making, just either average (as was suggested) or be selective as to which data set model appears to be more in line with your goals/data going forward. An understanding of the differences in the predictive classifications with respect to information content may also be a path to selecting a single model.