Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Say we have data where we have temperature, humidity, wind and rain for deciding whether we go and play tennis or not.
While selecting best feature to split upon initially, we use ID3 or Gini index. My question is why not use correlation ie when the features have maximum correlation with the output, split on that feature.
It seems obvious, but I can't find spit using correlation anywhere.
First of all, decision Trees can easily use categorical features with entropy and gini index, but (Pearson) correlation is defined for only numeric attributes. Using correlation with one-hot encoded versions of these categorical features also makes no sense. Secondly, In classification tasks, correlation between a, let's say binary output (e.g. $0,1$), and a numeric attribute may not make much sense as it normally does in regression problems. And, when it makes sense, it tries to capture the linear relationships by definition. (e.g. when $x=y^2$ and $x$ is symmetric for example, correlation is very low or zero)
For example, let's say we have a relationship $Y=1$ only when $X>0$, else $Y=0$. If we have small negative $X$, but comparably large positive $X$, as positive $X$'s move in positive/negative direction correlation changes because of their numeric values. But, actually this feature perfectly classifies the data.
While others are not perfect heuristics, correlation might be quite misleading when assessing the utility of the feature.
As already pointed out by @gunes correlation would only be available if both the dependent variable and the regressor or splitting variable is numeric. But in that situation it certainly makes sense to consider correlation because it will perform well in capturing monotonic associations. Hence, the CTree algorithm Hothorn et al. (2006) available in the R package partykit (and its predecessor "party") uses this criterion by default for the numeric-vs-numeric situation (suitably standardized). See this discussion for more details: What is the test statistics used for a conditional inference regression tree?
The formulation of the splitting criteria in CTree is also such that it generalizes to other association tests when one or both of the variables are not numeric. For example, in the categorical-vs-categorical situation a $\chi^2$ statistic is used.
If it is a potential concern that associations are not monotonic, one potential idea is to use a maximally-selected statistic over all possible split points. This is also possible, see: Explanation of different testtype and teststats in ctree in party package of R
