Why do the ensemble learners do well on regression/classification tasks? I was watching this short video on ensemble learners, and I am confused about why they tend to do better, and how is goodness measured. If the goodness means a low mean-squared error (MSE) as usual and if you are averaging the continuous-valued predictions from each learning method, then the MSE you get will apparently be the average of the MSEs from the individual methods, and which means ensemble has an average performance. So I don't understand why the error would decrease compared to the individual methods by the ensemble method. Apparently I am missing a big chunk here, and I appreciate any help.
 A: Your claim that the MSE of the average prediction is the average of the MSE is wrong, and this is the source of the confusion. In fact, it depends a lot on the estimators themselves. Estimators that are more diverse in some sense tend to have better performing ensemble.
Let $f_1, \ldots, f_m$ be the predictors. Then the MSE of the ensemble satisfies
$$
\mathbb{E}\left[ \left( \frac{1}{m} \sum_{i=1}^{m} f_i \left(X\right) - Y\right)^2 \right] = \frac{1}{m^2} \sum_{i=1}^{m} \mathbb{E}\left[ \left(  f_i \left(X\right) - Y\right)^2 \right] +  \frac{1}{m^2} \sum_{i \neq j}\mathbb{E}\left[ \left(  f_i \left(X\right) - Y\right)  \left(  f_j \left(X\right) - Y\right)\right]
$$
So, when the classifiers errors are uncorrelated, and there is no regressor that is much better than the rest, the ensemble outperforms the individual predictor, as in that case,
$$
MSE_{ens} = \frac{1}{m} \left(\frac{1}{m} \sum_{i=1}^{m} MSE_{f_i}\right)
$$
So, the MSE is much smaller than the average MSE of the predictors.
The first equation also suggests what cases we can't expect the ensemble to preform well. In fact, the best ensemble is usually a weighted average, with weights depending on the individual MSE and error correlation between the predictors.
For a more detailed discussion, you might consider reading chapter 4 of Combining Pattern Classifiers by Kuncheva.
A: Short answer (without technical stuff) would be https://www.quora.com/How-do-ensemble-methods-work-and-why-are-they-superior-to-individual-models
If you want to know exactly how then read about specific algorithms like random forest or gradient boost rather than overview of ensemble learning.
Good link for RF - https://jakevdp.github.io/PythonDataScienceHandbook/05.08-random-forests.html
A: 
If the goodness means a low mean-squared error (MSE) as usual and if
  you are averaging the continuous-valued predictions from each learning
  method,

A single model such as decision tree can easily give you a low MSE on the training set, but in ML you would like to measure your performance on non-training data. Unfortunately, most (all?) models produce bias while learning. By averaging many different models, you reduce bias and thus improve MSE on your non-training data set.
