0
$\begingroup$

I am trying to understand if one should standardise features for all models and when does it make sense to do so.

Is the below statement true? If yes, could you give a bit of explanation please.

Logistic regressions and tree-based algorithms such as decision trees, random forests and gradient boosting are not sensitive to the magnitude of variables. So standardization is not needed before fitting these kinds of models.

Found above in this link.

Is this the same thing as saying outliers don't impact tree based algorithms ?

$\endgroup$

1 Answer 1

1
$\begingroup$

The decision to standardize features first and foremost relies on the nature of the features themselves. Here are two reasons that I have come across in engineering:

  • features with different units cannot be taken into consideration as if they are equal, using something that takes away the units, e.g, z-score, takes away the unit and returns features that are comparable with each other.
  • features with exceptionally high or low values can cause the solvers ,such as gradient descent used in logistic regression, to struggle with computing meaningful fits. Reducing or increasing these values to something that the solvers can handle is very important.

Saying that outliers don't impact CARTs is also not accurate. Of course they do, outliers will cause splits in the tree that will be detrimental to the overall performance. Using ensemble learners will help, but I would say training a simple model on good data is always better than training a complex model with bad data.

Not to say that there are no cases where keeping the data as is don't exist. For example, ordinal variables can be kept as they are.

To scale data in python, check this page from scikit-learn.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.