4
$\begingroup$

I've read the paper on the XGBoost algorithm and one thing is not quite clear to me. The regularization term is defined as below where $w_{j}$ corresponds to the weight in end node $j$. For the regression case this would be the predicted value in the end node and for the classification case this would be either a $0$ or a $1$ I would suspect. Then the regularization term seems intuitive to me in the case of regression for penalizing large predictions however in the classification case it wouldn't add much as the $w_{j}$'s will always be a $0$ or a $1$ right?

enter image description here

$\endgroup$
2
  • 2
    $\begingroup$ You have a fundamental misunderstanding of boosting. Trees in the classification case are not decision trees fit to the responese, they are regression trees fit to the gradient of the loss function. $\endgroup$ Mar 7 '17 at 16:11
  • $\begingroup$ Related: stats.stackexchange.com/questions/353462/… $\endgroup$
    – Sycorax
    Jun 29 '18 at 14:26
4
$\begingroup$

XGBoost does not produce a decision tree/trees with leaf node values of 0 or 1.

Instead, it uses multiple regression trees with continuous value "weights" on its leaf nodes e.g. in the range 0 ~ 1. Hence, regularization is applied in the same manner as for similar regression based loss functions.

The tree boosting algorithm applies the trees additively i.e. sums the weights for all trees to arrive at the final value for the given input.

This continuous valued "score" needs to be interpreted by you as a class label by applying a cutoff, e.g. predicted class = 1 if $\hat y > 0.5 $, if cutoff = 0.5.

$\endgroup$
3
  • $\begingroup$ Since the trees are fit to the gradient of the loss, I don't think there is any constraint for them to lie in the [0, 1] range. But in spirit, this is correct. $\endgroup$ Mar 8 '17 at 16:55
  • $\begingroup$ Thanks for your answer @SandeepS.Sandhu! Could you elaborate on your comment @MatthewDrury? How are then the predicted values contained within the unit range? $\endgroup$
    – Andrew
    Mar 8 '17 at 23:36
  • 2
    $\begingroup$ Because the trees are not fit to predict the response. They are fit to predict the gradient of the response. The algorithm then replaces the gradient predictions with an ad-hoc computed prediction of the response itself in each terminal mode. It's important to understand this process for basic gradient boosting before worrying about XGBoost. $\endgroup$ Mar 8 '17 at 23:41
2
$\begingroup$

For classification:

Different from a decision tree for random forest where you can use majority of the class in the leaf code as the class for an observation, Boosting trees are totally different. In Boosting method, you have loss function as

$$ l(y_i,\hat{y_i})=y_i ln(1+e^{-\hat{y_i}}) + (1-y_i) ln(1+e^{\hat{y_i}}) $$

while $y_i$ is from your training data, $\hat{y_i}$ is from your boosting trees and equal to the sum of the leaf weights. If you have 10 weak learners (10 small trees) for your boosting, then you will have 10 weights ($w$) to sum over for a given X. Observations with 1's should have larger $\hat{y_i}$. But other than this, there is no physical meaning in $\hat{y_i}$.

We want to penalize for the sum of squared $w$'s to enforce each tree adds no substantial to the estimation, which matches the idea of each tree just serves as a weaker learner.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.