1
$\begingroup$

I have read articles mentioning that Random Forests and XGBoost does not require scaling or normalisation of features as decision trees identify thresholds or cut-points in the features. Is there any benefit to log transforming data in the event that my features comprise outliers, or does this not effect the thresholds or cut-points selected by Random Forests and XGBoost?

Are the preparation of continuous features for Random Forests and XGBoost the same for classification and regression models?

$\endgroup$

1 Answer 1

1
$\begingroup$

Basically, all 1-1 transformations (e.g. log-, sqrt-, exp-, square-transformation of positive numbers, standardization, adding constants, multiplying by a constant etc.) have no effect for random forest, XGBoost, LightGBM and similar algorithms that pick cut-points in variables. Non-1-1-transformations like taking the square of numbers that could be negative (in that case suddenly -1 and +1 get both mapped to 1 and treated as the same) on the other hand of course make a big difference.

That's very different from regression models, where things are easiest for you if you transform your feature so that it's relationship with the outcome is linear (after accounting for the other features). E.g. when you do a Poisson regression with a log-link-function for number of phone calls received in a day in a call center, then I'd probably first try log(phone calls last week) as a predictor rather than the untransformed number. Standardization tends to be useful for two reasons: 1) with extremely different scales between some features you can get numerical issues with regression, and 2) if you want regularization techniques to treat different variables similarly, it's easiest if they are on the same scale.

This doesn't mean, at all, that variable transformations are unimportant for XGBoost, RF and friends, it's just that it's usually more (possibly non-linear) transformations across multiple variables (and maybe records) that you may want. E.g. ratios such as when you have miles driven and fuel used for a car, then L per 100 km (or miles per gallon) could be a useful feature, which a human can understand as potentially important and provide so the model does not need to figure this out.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.