0
$\begingroup$

AdaBoost creates a large amount of weak learners, and gives each one prediction score/weight that will be used to combine the learners during evaluation/generalization.

But isn't the point to create different learners that are stronger at different examples? If so, weighting them basd on their overall error is counterproductive. There needs to be a dynamic, state-space framework to ensembling. Is this correct?

UPDATE:
I watched the suggested video, but I still do not feel like my question has been answered. If you combine a bunch of weak learners using unchanging coefficients, the error of each individually should be pretty high because they each only target specific aspects of the example distribution.

$\endgroup$
  • $\begingroup$ See this talk by Trevor Hastie. $\endgroup$ – Will May 7 '17 at 11:30
  • $\begingroup$ Thank you for the suggestion. I watched the video, but I still do not feel like my question has been answered. If you combine a bunch of weak learners using unchanging coefficients, the error of each individually should be pretty high because they each only target specific aspects of the example distribution. $\endgroup$ – Abraham Horowitz May 7 '17 at 13:29
  • $\begingroup$ Do you know gradient boosting? I feel like that algorithm is much easier to understand than AdaBoost. It's unfortunate that AdaBoost is still the first boosting algorithm that many learners encounter, it's both worse performing and more difficult to understand than gradient boosting. $\endgroup$ – Matthew Drury May 7 '17 at 19:48
  • $\begingroup$ I am familiar with it. It models the residuals as opposed to modeling a re-weighted distribution like AdaBoost does. $\endgroup$ – Abraham Horowitz May 7 '17 at 20:16
0
$\begingroup$

I think I figured it out. When reweighting the distribution, we multiply the example weights by a regularization parameter, which is the same coefficient that serves as the learner weight during model voting. It is computed as

$$ \lambda_j = \frac{1}{2}\ln\frac{\sum_{i: h_j(x_i) y_i = 1} w_j^i}{\sum_{i: h_j(x_i) y_i = -1} w_j^i} $$

in the case of binary classification. By multiplying the example weight vector by this, the algorithm prevents the next learner at iteration $j+1$ from overfitting to residuals, but still changes the distribution enough that many iterations, the individual learners will master the whole example distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.