1
$\begingroup$

I know gradient boosting use an iteration approach to finding a weak learner. But I am confused about the way to find weak learner, boosting PDF source

Question 1:

Why find the weak learner by the formula in the blue box?

Question 2:

What the meaning of the formula $w_ih(\mathbf x_i) $?

Question 3:

In the green box, I think the formula is to calculate the gradient, but how can I understand - before the formula?

Thanks for your help!

$\endgroup$
  • 1
    $\begingroup$ Quick edit, Boosting is not a greedy approach to find weak learner but start with a weak learner and iterate over it (over-sample mis-classifieds data, -> retrain -> update the weights -> repeat until convergence) $\endgroup$ – Nishad Jul 20 '18 at 3:45
  • 1
    $\begingroup$ The principle idea of gradient boosting is to make the weak learner at each iteration pointing to the direction of negative gadient of the current loss with respect to the current ensemble. Understanding this idea could make the equations straightforward. $\endgroup$ – doubllle Jul 20 '18 at 15:33
  • $\begingroup$ @Nishad Sorry, I have edited it! $\endgroup$ – Lei Chen Jul 21 '18 at 7:26
2
$\begingroup$

Why find the weak learner by the formula in the blue box?

You define a class of weak learners as part of your boosting algorithm. For example you can perform boosting on decision trees, on SVM, logistic regression etc.. All you need is a particular class of weak learner.

What the meaning of the formula wih(xi)?

For a particular iteration, you find the particular model (of the pre-defined class) that best fits your data, given the weights and parameters given in the formula

I think the formula is to calculate the gradient

In Gradient Boosting, ‘shortcomings’ (of existing weak learners) are identified by gradients. As @Doubllle says in comments: The principle idea of gradient boosting is to make the weak learner at each iteration pointing to the direction of negative gadient of the current loss with respect to the current ensemble."

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.