8
$\begingroup$

Can we generalize all ensemble methods by using voting? Do boosting methods also use voting to get the weak learners into the final model?

My understanding of the technique:

  • Boosting: Continuously adds in weak learner to boost the data points that were not correctly classified.
  • Ensemble technique: Uses multiple learners to obtain a better prediction than from one alone. This is explained in wikipedia.
$\endgroup$
6
$\begingroup$

Boosting can generally be understood as (weighted) voting

In the case of boosting, one of its inventors gives an affirmative answer in this introduction to AdaBoost (emphasis mine):

The final or combined hypothesis $H$ computes the sign of a weighted combination of weak hypotheses $$F(x) = \sum_{t=1}^T\alpha_th_t(x)$$ This is equivalent to saying that $H$ is computed as a weighted majority vote of the weak hypotheses $h_t$ where each is assigned weight $\alpha_t$. (In this chapter, we use the terms “hypothesis” and “classifier” interchangeably.)

So yes, the final model returned is a weighted vote of all the weak learners trained to that iteration. Likewise, you'll find this snippet on Wikipedia about boosting in general:

While boosting is not algorithmically constrained, most boosting algorithms consist of iteratively learning weak classifiers with respect to a distribution and adding them to a final strong classifier. When they are added, they are typically weighted in some way that is usually related to the weak learners' accuracy.

Also note the mention therein that the original boosting algorithms used a "majority." The notion of voting is pretty firmly baked into boosting: Its guiding principle is to improve an ensemble at each iteration by adding a new voter, then deciding how much weight to give each vote.

This same intuition carries for the example of gradient boosting: At each iteration $m$ we find a new learner $h_m$ fitted to pseudo-residuals, then optimize $\gamma_m$ to decide how much weight to give $h_m$'s "vote."

Extending to all ensemble methods runs into counterexamples

As it is, some would find that even the notion of weighting stretches the voting metaphor. When considering whether to extend this intuition to all ensemble learning methods, consider this snippet:

Ensembles combine multiple hypotheses to form a (hopefully) better hypothesis. The term ensemble is usually reserved for methods that generate multiple hypotheses using the same base learner.

And this one on the example ensemble method of stacking:

Stacking (sometimes called stacked generalization) involves training a learning algorithm to combine the predictions of several other learning algorithms. First, all of the other algorithms are trained using the available data, then a combiner algorithm is trained to make a final prediction using all the predictions of the other algorithms as additional inputs. If an arbitrary combiner algorithm is used, then stacking can theoretically represent any of the ensemble techniques described in this article, although in practice, a single-layer logistic regression model is often used as the combiner.

If you're defining ensemble methods to include stacking methods with an arbitrary combiner, you can construct methods that, in my view, stretch the notion of voting beyond its limit. It's difficult to see how a collection of weak learners combined via a decision tree or neural network can be viewed as "voting." (Leaving aside the also difficult question of when that method might prove practically useful.)

Some introductions describe ensembles and voting as synonymous; I'm not familiar enough with recent literature on these methods to say how these terms are generally applied recently, but I hope this answer gives an idea of how far the notion of voting extends.

$\endgroup$
  • $\begingroup$ very good answer $\endgroup$ – wogsland Mar 9 '17 at 1:10
  • $\begingroup$ Please explain how voting is done in gradient boosting machine. A weak learner is added at each iteration, so where is voting here. Can we generalize voting to be used in all boosting and also to all ensemble techniques? $\endgroup$ – pritywiz Mar 9 '17 at 5:00
  • 1
    $\begingroup$ To the first question, if you follow the description of gradient boosting here under "Algorithm," you'll find the final learner described as a weighted aim of weak learners. In essence, the voting metaphor is: At each iteration, you add a new voter focused on the pseudo-residuals, then optimize $\gamma_m$ to decide how much weight to give this new vote. $\endgroup$ – Sean Easter Mar 9 '17 at 13:41
  • 1
    $\begingroup$ To the second, I don't believe the voting metaphor carries water for all ensemble methods. If you read about stacking as described in common examples here, you'll find that an arbitrary combiner algorithm can be used, treating other learners' predictions as input. It's difficult to see how one could consider, say, a decision tree a voting mechanism among learners. Is that helpful? $\endgroup$ – Sean Easter Mar 9 '17 at 13:49
3
$\begingroup$

Boosting is different from bagging (voting). I do not see a way to interpret boosting as "voting" (see my edit for additional details).

  • Voting (especially majority vote) usually means combined decision from "separate / less correlated" week classifiers.

  • In boosting, we are building one classifier upon another. So, they are not "separate peers" but one is "less weaker than another".

My answers here gives boosting break down by iterations.

How does linear base leaner works in boosting? And how it works in xgboost library?

The example is trying to approximate a quadratic function by by boosting on decision stump.

  • First two plots are ground truth and boosting model after many iterations. They are contour plots. X and Y axis are two features and function value is represented by color.

enter image description here

  • Then I am showing first 4 iterations. You can see we are not averaging/voting 4 models, but enhance the model over each iterations.

enter image description here


After seeing another answer, I feel the answer to this question depends on how we define "voting". Do we consider weighted sum as voting ? If yes, then I think we still can say boosting can be generalized with voting.

$\endgroup$
  • $\begingroup$ I understand boosting as correctly explained by you, while in Adaboost we can say a weighted majority vote of all weak classifier is the final classifier, but it is not the same in case of GBM. So, we cannot generalize voting to be used in all ensemble techniques, isn't it? I am perplexed .. and precisely my confusion.. $\endgroup$ – pritywiz Mar 9 '17 at 9:12
  • 1
    $\begingroup$ @pritywiz I think another answer is also right. The word "voting" is not quite clear. The final form of GBM is still additive with different weights. Do we consider weighted sum = voting? $\endgroup$ – Haitao Du Mar 9 '17 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.