26
$\begingroup$

Doing research about the xgboost algorithm I went through the documentation.

I have heard that xgboost does not care much about the scale of the input features

In this approach trees are regularized using the complexity definition $$ \Omega(f) = \gamma T + \frac12 \lambda \sum_{j=1}^T w_j^2 $$ where $\gamma$ and $\lambda$ are parameters, $T$ is the number of terminal leaves and $w_j$ is the score in each leaf.

So does this not make it important to scale the features before feeding into xgboost? $\sum_{j=1}^T w_j^2$ term in the regularization part of the cost function is directly influenced by the scale of the features

$\endgroup$

1 Answer 1

38
$\begingroup$

XGBoost is not sensitive to monotonic transformations of its features for the same reason that decision trees and random forests are not: the model only needs to pick "cut points" on features to split a node. Splits are not sensitive to monotonic transformations: defining a split on one scale has a corresponding split on the transformed scale.

Your confusion stems from misunderstanding $w$. In the section "Model Complexity," the author writes

Here $w$ is the vector of scores on leaves...

The score measures the weight of the leaf. See the diagram in the "Tree Ensemble" section; the author labels the number below the leaf as the "score."

score diagram

The score is also defined more precisely in the paragraph preceding your expression for $\Omega(f)$:

We need to define the complexity of the tree $\Omega(f)$. In order to do so, let us first refine the definition of the tree $f(x)$ as $$f_t(x)=w_{q(x)}, w \in R^T, q:R^d \to {1,2,\dots,T}.$$ Here $w$ is the vector of scores on leaves, $q$ is a function assigning each data point to the corresponding leaf, and $T$ is the number of leaves.

What this expression is saying is that $q$ is a partitioning function of $R^d$, and $w$ is the weight associated with each partition. Partitioning $R^d$ can be done with coordinate-aligned splits, and coordinate-aligned splits are decision trees.

The meaning of $w$ is that it is a "weight" chosen so that the loss of the ensemble with the new tree is lower than the loss of the ensemble without the new tree. This is described in "The Structure Score" section of the documentation. The score for a leaf $j$ is given by

$$ w_j^* = \frac{G_j}{H_j + \lambda} $$

where $G_j$ and $H_j$ are the sums of functions of the partial derivatives of the loss function wrt the prediction for tree $t-1$ for the samples in the $j$th leaf. (See "Additive Training" for details.)

$\endgroup$
4
  • $\begingroup$ Thank you for answer. Let me be sure that i got this: Let us assume a regression problem - predicting housing prices. Loss is SSE. Score in each leaf is going to be the mean of target variable in that leaf. Now i center and standardize the housing prices (Millions to around 0). Now you see the regularization is changing massively and will completely dominate the loss contribution by the number of leaves. But i am guessing that is what $\lambda$ and $\gamma$ are for. Am i right? $\endgroup$ Commented Jun 28, 2018 at 12:04
  • 2
    $\begingroup$ No, that's not correct. The scores in the leafs is given by the gradient boosting procedure. I've updated my answer to include some information about how that works. The boosting procedure only checks for the presence/absence of a sample in a leaf, and doesn't care about the scale of the features. Nor do tree induction algorithms care about the scale of the features. $\endgroup$
    – Sycorax
    Commented Jun 28, 2018 at 14:39
  • 1
    $\begingroup$ This may be the same question as @MiloMinderbinder, but what I wonder is: since $G_j$ is the summed up gradient, which are the residuals for squared loss, it should drastically change upon standardization. $H_j$, the sum of the hessian, is just the number of observations for squared loss, which remains the same upon standardization. Therefore, given the formula above, $w_j$ should drastically change upon standardization of the data while keeping the same $\lambda$ & $\gamma$, see here. After some simulations, this doesn't seem to be the case. Why? $\endgroup$
    – PaulG
    Commented Mar 16, 2021 at 15:23
  • 4
    $\begingroup$ @PaulG This question asks about scaling the features of the model. Your comment seems to ask about scaling the target of the model. Milo's comment also asks about the target, but relies upon a mistaken understanding of predictions. Since comment boxes are small, I think it's best to ask a new question so you can more completely lay out what your thinking is and what you know and what you'd like to find out. $\endgroup$
    – Sycorax
    Commented Mar 16, 2021 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.