Can we generalize all ensemble methods by using voting? Do boosting methods also use voting to get the weak learners into the final model?
My understanding of the technique:
In the case of boosting, one of its inventors gives an affirmative answer in this introduction to AdaBoost (emphasis mine):
The final or combined hypothesis $H$ computes the sign of a weighted combination of weak hypotheses $$F(x) = \sum_{t=1}^T\alpha_th_t(x)$$ This is equivalent to saying that $H$ is computed as a weighted majority vote of the weak hypotheses $h_t$ where each is assigned weight $\alpha_t$. (In this chapter, we use the terms “hypothesis” and “classifier” interchangeably.)
So yes, the final model returned is a weighted vote of all the weak learners trained to that iteration. Likewise, you'll find this snippet on Wikipedia about boosting in general:
While boosting is not algorithmically constrained, most boosting algorithms consist of iteratively learning weak classifiers with respect to a distribution and adding them to a final strong classifier. When they are added, they are typically weighted in some way that is usually related to the weak learners' accuracy.
Also note the mention therein that the original boosting algorithms used a "majority." The notion of voting is pretty firmly baked into boosting: Its guiding principle is to improve an ensemble at each iteration by adding a new voter, then deciding how much weight to give each vote.
This same intuition carries for the example of gradient boosting: At each iteration $m$ we find a new learner $h_m$ fitted to pseudo-residuals, then optimize $\gamma_m$ to decide how much weight to give $h_m$'s "vote."
As it is, some would find that even the notion of weighting stretches the voting metaphor. When considering whether to extend this intuition to all ensemble learning methods, consider this snippet:
Ensembles combine multiple hypotheses to form a (hopefully) better hypothesis. The term ensemble is usually reserved for methods that generate multiple hypotheses using the same base learner.
And this one on the example ensemble method of stacking:
Stacking (sometimes called stacked generalization) involves training a learning algorithm to combine the predictions of several other learning algorithms. First, all of the other algorithms are trained using the available data, then a combiner algorithm is trained to make a final prediction using all the predictions of the other algorithms as additional inputs. If an arbitrary combiner algorithm is used, then stacking can theoretically represent any of the ensemble techniques described in this article, although in practice, a single-layer logistic regression model is often used as the combiner.
If you're defining ensemble methods to include stacking methods with an arbitrary combiner, you can construct methods that, in my view, stretch the notion of voting beyond its limit. It's difficult to see how a collection of weak learners combined via a decision tree or neural network can be viewed as "voting." (Leaving aside the also difficult question of when that method might prove practically useful.)
Some introductions describe ensembles and voting as synonymous; I'm not familiar enough with recent literature on these methods to say how these terms are generally applied recently, but I hope this answer gives an idea of how far the notion of voting extends.
Boosting is different from bagging (voting). I do not see a way to interpret boosting as "voting" (see my edit for additional details).
Voting (especially majority vote) usually means combined decision from "separate / less correlated" week classifiers.
In boosting, we are building one classifier upon another. So, they are not "separate peers" but one is "less weaker than another".
My answers here gives boosting break down by iterations.
How does linear base leaner works in boosting? And how it works in xgboost library?
The example is trying to approximate a quadratic function by by boosting on decision stump.
After seeing another answer, I feel the answer to this question depends on how we define "voting". Do we consider weighted sum as voting ? If yes, then I think we still can say boosting can be generalized with voting.
