Diversity between classifiers in ensemble learning

Question

According to Wikipedia, Ensemble of models tends to yield better results when there is a significant diversity among the models. Many ensemble methods, therefore, seek to promote diversity among the models they combine. Using a variety of strong learning algorithms has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity. I have two questions please regarding this paragraph:

1. Could you please explain the meaning of diversity with simple example?
2. In ensemble classifier, do we combine (using for example, majority voting) weak classifiers or strong classifiers, or a combination of both weak and strong classifiers?

Answer 1

Your first question: This is probably intentionally formulated very vaguely on Wikipedia. But, as an example, you could think of Bayesian predictions as the weighted average of many models which differ continuously in their parameters. And often the weights are only large in a confined region, so the relevant models in this ensemble differ only by a small change in the parameters, which could qualify for small diversity in the models. A larger diversity would be obtained if you were to fit completely different models, e.g. a decision tree and an SVM.

Your second question: Following your citation of Wikipedia, a combination of strong classifiers would be preferable. But there are many examples where an ensemble of weak classifiers can obtain excellent results, like e.g. random forests or gradient boosting machines, provided the ensembles contain sufficiently many models. The performance is also very much dependent on how exactly you do the ensembling. Ordinary majority voting is a less sophisticated method, but, given enough models, might provide good results.

Most of the time the rule of thumb is: the more models, the better, no matter whether they are weak or strong.

Answer 2

Ensemble of models tends to yield better results when there is a significant diversity among the models

Let me rephrase this: iff there is high variance between the submodels (or their predictions), ensemble methods help.
(This is very similar to what we routinely do e.g. with measurements iff the dominant factor for measurement uncertainty is variance: we then average multiple measurements, or go for the diagnosis of a panel of sensory experts, etc.)

The conclusion from this is that the submodels should be representative for the respective variance. (But it doesn't help to introduce arbitrary variance that isn't relevant for the model/data/application-combination)

Ensembles cannot help lowering the part of prediction error that is caused by bias.
Thus, the submodels of the ensemble should not have too much bias, but their variance is of less concern - since we plan to "average it away".

Now, a strong model may have achieved its good performance for a given (in particular: limited) data set by finding good bias-variance tradeoff for the single model prediction. This compromise may be sub-optimal for the ensemble submodels, in the sense that the ensemble may be able to achieve even better performace with (somewhat) weaker sub-models that reduce bias by accepting more variance compared to the strong model. Promoting variance in the sense of moving from such strong submodels to the corresponding weaker submodels does make sense. In other words, the optimal (sub)model complexity for a given application, data set, and training algorithm is different whether the model will be used individually or in an ensemble.

Another conclusion is: if you move from the optimal somewhat-weaker submodels to still weaker models with even more variance - that however isn't offset by lower bias - then that additional variance won't help the ensemble.

---

And lastly, if the strong model is strong by having both low bias and low variance, no ensemble is needed.