Standardizing inputs for CART

Question

I know I do not need to standardize the predictor variables before applying CART but would there be any adverse effects to doing it anyway? I'm comparing CART to a linear regression where I did standardize the inputs and for the sake of consistency I'd like to do the same for CART, but could it be detrimental to my results? I'm not really sure how CART goes about finding the splitting point at each node (I can't find a copy of the original paper and most of what I've read just skips over that detail) but I would think that standardizing the inputs would actually help speed up that process. Does anyone have any intuition in this regard?

Answer 1

My answer at this stage is based on my intuitive understanding of how CART works, so take it face value.

Standardization does not add or subtract information contained in a given variable and does not distort its relationship to a target variable. For example, if you had a variable "age" which was a predictor for "purchase car". By changing age to (age - mean / sd ) is not going to change its relationship to purchase car, it merely maps it to a new space.

When CART looks for the best splits, it going to use entropy or gini to calculate information gain, this is not dependent on the scale of your predictor variable, rather on the resultant purity of the variable "purchase car".

So long story cut short, standardization should have no influence on your final solution. To verify this, simply run your model twice. Once on the standardized version of data and once with the original. If all goes well, you should see the same variables and order of selection, just different cut off points.

Edit: I've appended two images for your reference for a tree with and without standardized inputs.

Answer 2

I will try to give a formal proof here to clarify things.

Step 1

We denote with $x = {x_1, x_2, .., x_n}$ a vector, and with $var(x)=\frac{1}{n}\sum_{i=1}^{n}(x_i-\frac{1}{n}\sum_{j=1}^{n}x_j)^2$ the biased sample variance. We denote with $y = \frac{x-m}{sd}$ the scaled version.

We work out a formula for $var(y)$ in terms of $var(x),m,sd$.

$$var(y)=\frac{1}{n}\sum_{i=1}^{n}(y_i-\frac{1}{n}\sum_{j=1}^{n}y_j)^2 = \frac{1}{n}\sum_{i=1}^{n}(\frac{x_i-m}{sd}-\frac{1}{n}\sum_{j=1}^{n}\frac{x_j-m}{sd})^2 $$ $$= \frac{1}{n sd^2}\sum_{i=1}^{n}(x_i-m-\frac{1}{n}\sum_{j=1}^{n}(x_j-m))^2$$ $$ = \frac{1}{n sd^2}\sum_{i=1}^{n}(x_i-m-\frac{1}{n}\sum_{j=1}^{n}x_j+\frac{1}{n}\sum_{j=1}^{n}m)^2 $$ $$ = \frac{1}{n sd^2}\sum_{i=1}^{n}(x_i-m-\frac{1}{n}\sum_{j=1}^{n}x_j+m)^2 = \frac{1}{n sd^2}\sum_{i=1}^{n}(x_i-\frac{1}{n}\sum_{j=1}^{n}x_j)^2 $$ $$ = \frac{1}{sd^2}var(x) $$ So we know that the variance of the scaled version is proportional with the variance of the unscaled version. $$var(y)=\frac{1}{sd^2}var(x)$$

Step 2

For CART regression we know that the split test is evaluated with a formula like $var(x_{left})+var(x_{right})$, where $x_{left}$ consists of all observation values of the target variable which are less than or equal some threshold value on the test variable.

In order to show that various tests preserves the same order after scaling I think there is enough to say something like:

For any given two binary splits of the values of the target variables, if the evaluation function for the first split is less than the evaluation function for the second split on the unscaled variable implies that the evaluation function for the first split is less than evaluation function for the second split on scaled variable.

In plain English comparison between splits is the same for scaled and unscaled variables.

Demonstration is trivial since the evaluation function is $split(x_{left},x_{x_right})=var(x_{left})+var(x_{right})$, and we know that variance for unscaled version is proportional with variance for scaled version and this property is preserved on addition.

Final comments

Based on intuition, I agreed with @ArunJose from the begining. However, I wanted a solid proof. So, no offense, @ArunJose, I just wanted to be sure.