I was watching this short video on ensemble learners, and I am confused about why they tend to do better, and how is goodness measured. If the goodness means a low mean-squared error (MSE) as usual and if you are averaging the continuous-valued predictions from each learning method, then the MSE you get will apparently be the average of the MSEs from the individual methods, and which means ensemble has an average performance. So I don't understand why the error would decrease compared to the individual methods by the ensemble method. Apparently I am missing a big chunk here, and I appreciate any help.
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
Your claim that the MSE of the average prediction is the average of the MSE is wrong, and this is the source of the confusion. In fact, it depends a lot on the estimators themselves. Estimators that are more diverse in some sense tend to have better performing ensemble.
Let $f_1, \ldots, f_m$ be the predictors. Then the MSE of the ensemble satisfies $$ \mathbb{E}\left[ \left( \frac{1}{m} \sum_{i=1}^{m} f_i \left(X\right) - Y\right)^2 \right] = \frac{1}{m^2} \sum_{i=1}^{m} \mathbb{E}\left[ \left( f_i \left(X\right) - Y\right)^2 \right] + \frac{1}{m^2} \sum_{i \neq j}\mathbb{E}\left[ \left( f_i \left(X\right) - Y\right) \left( f_j \left(X\right) - Y\right)\right] $$
So, when the classifiers errors are uncorrelated, and there is no regressor that is much better than the rest, the ensemble outperforms the individual predictor, as in that case, $$ MSE_{ens} = \frac{1}{m} \left(\frac{1}{m} \sum_{i=1}^{m} MSE_{f_i}\right) $$ So, the MSE is much smaller than the average MSE of the predictors.
The first equation also suggests what cases we can't expect the ensemble to preform well. In fact, the best ensemble is usually a weighted average, with weights depending on the individual MSE and error correlation between the predictors.
For a more detailed discussion, you might consider reading chapter 4 of Combining Pattern Classifiers by Kuncheva.
$\begingroup$
$\endgroup$
2
Short answer (without technical stuff) would be https://www.quora.com/How-do-ensemble-methods-work-and-why-are-they-superior-to-individual-models
If you want to know exactly how then read about specific algorithms like random forest or gradient boost rather than overview of ensemble learning.
Good link for RF - https://jakevdp.github.io/PythonDataScienceHandbook/05.08-random-forests.html
-
$\begingroup$ Thanks, I have already read an overview, but I need a more technical answer. Do you have any response in terms of the MSE example I gave? Let's say the true outcome value for a sample is 0.3. And say a trained linear regression model predicted 0.5, and decision tree regression predicted 0.4. If we take an ensemble of these methods, what we will get is 0.45 as the prediction. And as expected, the error now is higher than decision tree's error and lower than linear regression's error. I just don't understand the mathematical evidence that ensemble does better than each of the individual methods. $\endgroup$– user5054Jul 26, 2018 at 4:57
-
$\begingroup$ @user5054 Check this - mlwave.com/kaggle-ensembling-guide. Note that ensemble won't always be better. $\endgroup$– wololoJul 26, 2018 at 8:06
$\begingroup$
$\endgroup$
7
If the goodness means a low mean-squared error (MSE) as usual and if you are averaging the continuous-valued predictions from each learning method,
A single model such as decision tree can easily give you a low MSE on the training set, but in ML you would like to measure your performance on non-training data. Unfortunately, most (all?) models produce bias while learning. By averaging many different models, you reduce bias and thus improve MSE on your non-training data set.
-
$\begingroup$ The prediction examples in my comment to the reply above are on the test (non-training) data. $\endgroup$– user5054Jul 26, 2018 at 6:34
-
$\begingroup$ @user5054 Good, so my answer is 100% perfect fit for your question. Do you have anymore you want to know? $\endgroup$– ABCDJul 26, 2018 at 6:42
-
1$\begingroup$ This is not completely correct. Ensembling doesn't always reduce bias. For example, in random forests, bias of the ensemble is actually slightly greater than for an individual tree, and ensembling helps by reducing variance. $\endgroup$ Jul 26, 2018 at 23:19
-
1$\begingroup$ Right, on average. Expected error decomposes into bias, variance, and irreducible error. You said that ensembling works by reducing bias, which is true for some methods. But it's untrue (even on average) for others (which work by reducing variance instead). $\endgroup$ Jul 27, 2018 at 3:38
-
1$\begingroup$ The term bias has a very specific meaning in machine learning and statistics. Morover, this usage is part of the canonical explanation for ensemble methods (i.e. the bias-variance decomposition). If you use standard statistical terminology in another way it's misleading and you may cause others to have the wrong idea. I'm not convinced that the english dictionary definition is correct here either. But, if you think you have the right idea, may I suggest editing your post to clarify what you mean? $\endgroup$ Jul 27, 2018 at 7:14