# Defining Diversity in Ensemble Learning

I have a few questions regarding on how diversity is defined since I've seen differing definitions in different papers.

In the paper "Measures of Diversity in Classifier Ensembles and their Relationship with Ensemble Accuracy" by Ludmila I. Kuncheva (2003) I saw this definition:

When classifiers output class labels, the classification error can be
decomposed into bias and variance terms (also called ‘spread’) (Bauer & Kohavi, 1999; Breiman, 1999; Kohavi & Wolpert, 1996) or into bias and spread > terms. In both cases the second term can be taken as the diversity of the ensemble.

However in other places I've seen that the diversity is based on how strongly correlated are the base learners. The less correlated they are the more diverse they are.

Or is there no difference since $Corr(X,Y) = \frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}$. The higher the variance the smaller the correlation and the larger the diversity will be(?)

As well, a high level definition I've heard too is that diversity measures the number of coincident errors committed by classifiers in an ensemble. Would this be a true statement still if the diversity was really the variance or correlation among classifiers?

Ensemble diversity, that is, the difference among the individual learners, is a fundamental issue in ensemble methods. diversity is crucial to ensemble performance. Generating diverse individual learners, however is not easy. The major obstacle lies in the fact that the individual learners are trained for the same task from the same training data, and thus they are usually highly correlated. individual learners should be accurate and diverse. Combining only accurate learners is often worse than combining some accurate ones together with some relatively weak ones, since complementarity is more important than pure accuracy. the success of ensemble learning lies in achieving a good tradeoff between the individual performance and diversity.

Reference : Zhou Zhihua (2012). Ensemble Methods: Foundations and Algorithms. Chapman and Hall/CRC. ISBN 978-1-439-83003-1.

Ensemble diversity is a measure of variation in the ensemble members, with respect to changes in their training data, or other random variables involved in their construction. It is arguable that it can only be referred to as "diversity" when the measure is independent of the target variable y. The notion of correlation being high/low/independent is the special case that applies only for the squared loss function.

In the more general case of other losses, it turns out the formulation of diversity is only possible [1] when the ensemble combination is the so-called "centroid" combiner rule, derived directly from the loss function. In this case, the formalisation of diversity is exactly the same as in the bias-variance decomposition, but the diversity is an extra dimension created by their agreements/disagreements. It turns out that for 0-1 loss (the loss of most interest with ensemble classifiers) such a decomposition is provably not possible. To quote the paper referenced below:

"Overall, we argue that diversity is best understood as a measure of model fit, in precisely the same sense as bias and variance, but accounting for statistical dependencies between ensemble members. With single models, we have a bias/variance trade-off. With an ensemble we have a bias/variance/diversity trade-off—varying both with individual model capacity, and similarities between model predictions."

[1] Wood et al, 2023. "A Unified Theory of Diversity in Ensemble Learning arXiv:2301.03962v1, https://arxiv.org/pdf/2301.03962.pdf

Full disclosure: I'm an author.

• Welcome to CV, Gavin. May 19, 2023 at 9:00
• "...can only be referred to as "diversity" when the measure is independent of the target variable $y$" -- I've been wondering why that is, would you mind elaborating a bit? See also my question here. Jun 9, 2023 at 12:16
• Have elaborated as reply to your question... Jun 19, 2023 at 10:14