Theoretical explanation of the superior performance of ensemble machine learning algorithms

Question

In my experience and I believe also in many others' experiences, ensemble models such as random forests and boosting give superior results compared to other models such as linear/logistic regression or single decision trees (I remember some experimental work which also confirms this but I could not find now). Of course there might be cases where linear regression is best, for example, when the underlying relationship is indeed linear. But ensemble models in general have better performance.

What is the explanation of this situation from a machine learning theory perspective? Are there some proofs which show that they have better generalization power? Any pointers to resources will also be appreciated.

Answer 1

We know that error in a certain model can be composited from bias and variance. A too complex model has low bias but large variance, while a too simple model has low variance but large bias, both leading to a high error but for two different reasons. As a result, two different ways to solve the problem come into people's mind (at leat to Breiman's mind), variance reduction for a complex model, or bias reduction for a simple model, which refers to bagging and boosting.

Answer 2

Even though bagging and boosting are both ensemble methods they are conceptually different. While I'm not aware of theoretical aspects of boosting, bagging ensemble methods are mainly used to reduce the variance in predictions of individual models.

Consider, e.g., a random variable following a Gaussian distribution, i.e., $x \sim \mathcal{N}(0,\sigma)$. And consider a new random variable that is defined as the average of $N$ elements sampled from $\mathcal{N}$, i.e. $X = \frac{1}{N} \sum_{i=1}^N x_i$ where $x_i \sim \mathcal{N} \text{ for } i=1,\dots N$. Then, one can see that the mean of the random variable $X$ is $0$ as well. But the variance changes:

$$ \text{Var}(X) = \text{Var}(\frac{1}{N} \sum_{i=1}^N x_i) = \frac{1}{N^2} \sum_{i=1}^N \text{Var}(x_i) = \frac{1}{N^2} N \sigma = \frac{\sigma}{N} $$ Where Bienaymé formula was used in the second equality as $x_i$ are uncorrelated. That is, when combining several inference models to make a prediction, one can have a final model with a reduced variance. To see the influence on the generalization error you can use the decomposition of this error into bias and variance.

As for references, an introduction to statistical learning by James et al. touches on this topic in chapter 8.2 Bagging, random forests and boosting. But not with many details.