2
$\begingroup$

According to Wikipedia, Ensemble of models tends to yield better results when there is a significant diversity among the models. Many ensemble methods, therefore, seek to promote diversity among the models they combine. Using a variety of strong learning algorithms has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity. I have two questions please regarding this paragraph:

  1. Could you please explain the meaning of diversity with simple example?
  2. In ensemble classifier, do we combine (using for example, majority voting) weak classifiers or strong classifiers, or a combination of both weak and strong classifiers?
$\endgroup$

2 Answers 2

1
$\begingroup$

Your first question: This is probably intentionally formulated very vaguely on Wikipedia. But, as an example, you could think of Bayesian predictions as the weighted average of many models which differ continuously in their parameters. And often the weights are only large in a confined region, so the relevant models in this ensemble differ only by a small change in the parameters, which could qualify for small diversity in the models. A larger diversity would be obtained if you were to fit completely different models, e.g. a decision tree and an SVM.

Your second question: Following your citation of Wikipedia, a combination of strong classifiers would be preferable. But there are many examples where an ensemble of weak classifiers can obtain excellent results, like e.g. random forests or gradient boosting machines, provided the ensembles contain sufficiently many models. The performance is also very much dependent on how exactly you do the ensembling. Ordinary majority voting is a less sophisticated method, but, given enough models, might provide good results.

Most of the time the rule of thumb is: the more models, the better, no matter whether they are weak or strong.

$\endgroup$
4
  • $\begingroup$ For a cancer disease classification problem, I have employed two pre-trained fine-tuned deep networks and obtained a classification accuracy of about 97% from each of them. I have also designed my own convolutional network from scratch and obtained a classification accuracy of about 95%. Then, I have applied decision level fusion using majority voting between the three networks and obtained an enhanced accuracy of about 99%. Generally, weak learners disagree frequently and ensembling a lot of learners improve the performance. $\endgroup$
    – Noha
    May 27, 2022 at 11:45
  • $\begingroup$ But, strong learners that individually have high classification performance, tend to disagree on “difficult to classify” samples. Now since adding my designed network increased the performance of the ensemble, that means that it made correct prediction on those “difficult to classify” samples. Could you please help me interpreting how it is performing well on difficult samples, although it is the lowest in accuracy when operating alone? $\endgroup$
    – Noha
    May 27, 2022 at 11:45
  • $\begingroup$ When both the pre-trained fine-tuned networks agree on a label, then the prediction of the designed network is redundant since the vote will be dominated by the prediction of the two fine-tuned networks. This implies the only situation when the the designed from scratch network will influence the decision is when the two fine-tuned networks disagree. Upon checking the results, I noticed that this situation has occurred with many samples. $\endgroup$
    – Noha
    May 27, 2022 at 11:46
  • 1
    $\begingroup$ @Noha Those are interesting observations. Your designed network seems to be a useful tiebreaker. Note, however, that this is not an exact science, and most of ensemble design is just trial and error. And you seem to be quite successful :) $\endgroup$
    – frank
    May 27, 2022 at 13:54
2
$\begingroup$

Ensemble of models tends to yield better results when there is a significant diversity among the models

Let me rephrase this: iff there is high variance between the submodels (or their predictions), ensemble methods help.
(This is very similar to what we routinely do e.g. with measurements iff the dominant factor for measurement uncertainty is variance: we then average multiple measurements, or go for the diagnosis of a panel of sensory experts, etc.)

The conclusion from this is that the submodels should be representative for the respective variance. (But it doesn't help to introduce arbitrary variance that isn't relevant for the model/data/application-combination)

Ensembles cannot help lowering the part of prediction error that is caused by bias.
Thus, the submodels of the ensemble should not have too much bias, but their variance is of less concern - since we plan to "average it away".

Now, a strong model may have achieved its good performance for a given (in particular: limited) data set by finding good bias-variance tradeoff for the single model prediction. This compromise may be sub-optimal for the ensemble submodels, in the sense that the ensemble may be able to achieve even better performace with (somewhat) weaker sub-models that reduce bias by accepting more variance compared to the strong model. Promoting variance in the sense of moving from such strong submodels to the corresponding weaker submodels does make sense. In other words, the optimal (sub)model complexity for a given application, data set, and training algorithm is different whether the model will be used individually or in an ensemble.

Another conclusion is: if you move from the optimal somewhat-weaker submodels to still weaker models with even more variance - that however isn't offset by lower bias - then that additional variance won't help the ensemble.


And lastly, if the strong model is strong by having both low bias and low variance, no ensemble is needed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.