For all tree-based models like xgboost, lightgbm, random forest, they do not require feature scaling given the nature in which they compute their splits. However, when you perform regularization, one generally has to scale their features to account for any improper penalization. Does this apply to these types of models like xgboost and lightgbm, when you have input parameters to tune L1 and L2 regularization?
$\begingroup$
$\endgroup$
asked Dec 30, 2021 at 12:21
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
No, L1 or L2 regularization do not require any feature scaling for tree ensembles. In linear regression the regularization terms (here $\alpha$ and $\lambda$) scale a penalty based on the L1 or (square of) L2 norm of the coefficient vector. This makes standardization important because otherwise different features will be affected by the regularization differently depending on their units.
For Xgboost and Lightgbm the L1 and L2 regularization terms scale a penalty based on the L1 or (square of) L2 norms of a node's output value (each node output is scaled separately but with the same $\alpha$ or $\lambda$, this is just absolute or square value of a scalar). They show up like this in the calculation of a node's contribution to the prediction:
$$\frac{-G \pm \alpha}{H + \lambda}$$
Where $G$ (similarly $H$) is the sum of first derivatives (similarly second derivatives) of your loss function, over all data points in the node, evaluated at the prior stage prediction for each point. Note that the numerical values of the features don't enter in here at all.
I'm not familiar with regularization implementation like this for Random Forest.
