1
$\begingroup$

I know gradient boosting use an iteration approach to finding a weak learner. But I am confused about the way to find weak learner, boosting PDF source

Question 1:

Why find the weak learner by the formula in the blue box?

Question 2:

What the meaning of the formula $w_ih(\mathbf x_i) $?

Question 3:

In the green box, I think the formula is to calculate the gradient, but how can I understand - before the formula?

Thanks for your help!

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Quick edit, Boosting is not a greedy approach to find weak learner but start with a weak learner and iterate over it (over-sample mis-classifieds data, -> retrain -> update the weights -> repeat until convergence) $\endgroup$
    – wololo
    Jul 20, 2018 at 3:45
  • 1
    $\begingroup$ The principle idea of gradient boosting is to make the weak learner at each iteration pointing to the direction of negative gadient of the current loss with respect to the current ensemble. Understanding this idea could make the equations straightforward. $\endgroup$
    – doubllle
    Jul 20, 2018 at 15:33
  • $\begingroup$ @Nishad Sorry, I have edited it! $\endgroup$
    – Lei Chen
    Jul 21, 2018 at 7:26

1 Answer 1

Reset to default
2
$\begingroup$

Why find the weak learner by the formula in the blue box?

You define a class of weak learners as part of your boosting algorithm. For example you can perform boosting on decision trees, on SVM, logistic regression etc.. All you need is a particular class of weak learner.

What the meaning of the formula wih(xi)?

For a particular iteration, you find the particular model (of the pre-defined class) that best fits your data, given the weights and parameters given in the formula

I think the formula is to calculate the gradient

In Gradient Boosting, ‘shortcomings’ (of existing weak learners) are identified by gradients. As @Doubllle says in comments: The principle idea of gradient boosting is to make the weak learner at each iteration pointing to the direction of negative gadient of the current loss with respect to the current ensemble."

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.