# Theoretical explanation of the superior performance of ensemble machine learning algorithms

In my experience and I believe also in many others' experiences, ensemble models such as random forests and boosting give superior results compared to other models such as linear/logistic regression or single decision trees (I remember some experimental work which also confirms this but I could not find now). Of course there might be cases where linear regression is best, for example, when the underlying relationship is indeed linear. But ensemble models in general have better performance.

What is the explanation of this situation from a machine learning theory perspective? Are there some proofs which show that they have better generalization power? Any pointers to resources will also be appreciated.

Consider, e.g., a random variable following a Gaussian distribution, i.e., $$x \sim \mathcal{N}(0,\sigma)$$. And consider a new random variable that is defined as the average of $$N$$ elements sampled from $$\mathcal{N}$$, i.e. $$X = \frac{1}{N} \sum_{i=1}^N x_i$$ where $$x_i \sim \mathcal{N} \text{ for } i=1,\dots N$$. Then, one can see that the mean of the random variable $$X$$ is $$0$$ as well. But the variance changes:
$$\text{Var}(X) = \text{Var}(\frac{1}{N} \sum_{i=1}^N x_i) = \frac{1}{N^2} \sum_{i=1}^N \text{Var}(x_i) = \frac{1}{N^2} N \sigma = \frac{\sigma}{N}$$ Where Bienaymé formula was used in the second equality as $$x_i$$ are uncorrelated. That is, when combining several inference models to make a prediction, one can have a final model with a reduced variance. To see the influence on the generalization error you can use the decomposition of this error into bias and variance.