XGBoost - Weights in classifcation tree 0 or 1?

Question

I've read the paper on the XGBoost algorithm and one thing is not quite clear to me. The regularization term is defined as below where $w_{j}$ corresponds to the weight in end node $j$. For the regression case this would be the predicted value in the end node and for the classification case this would be either a $0$ or a $1$ I would suspect. Then the regularization term seems intuitive to me in the case of regression for penalizing large predictions however in the classification case it wouldn't add much as the $w_{j}$'s will always be a $0$ or a $1$ right?

Answer 1

XGBoost does not produce a decision tree/trees with leaf node values of 0 or 1.

Instead, it uses multiple regression trees with continuous value "weights" on its leaf nodes e.g. in the range 0 ~ 1. Hence, regularization is applied in the same manner as for similar regression based loss functions.

The tree boosting algorithm applies the trees additively i.e. sums the weights for all trees to arrive at the final value for the given input.

This continuous valued "score" needs to be interpreted by you as a class label by applying a cutoff, e.g. predicted class = 1 if $\hat y > 0.5 $, if cutoff = 0.5.

Answer 2

For classification:

Different from a decision tree for random forest where you can use majority of the class in the leaf code as the class for an observation, Boosting trees are totally different. In Boosting method, you have loss function as

$$ l(y_i,\hat{y_i})=y_i ln(1+e^{-\hat{y_i}}) + (1-y_i) ln(1+e^{\hat{y_i}}) $$

while $y_i$ is from your training data, $\hat{y_i}$ is from your boosting trees and equal to the sum of the leaf weights. If you have 10 weak learners (10 small trees) for your boosting, then you will have 10 weights ($w$) to sum over for a given X. Observations with 1's should have larger $\hat{y_i}$. But other than this, there is no physical meaning in $\hat{y_i}$.

We want to penalize for the sum of squared $w$'s to enforce each tree adds no substantial to the estimation, which matches the idea of each tree just serves as a weaker learner.