1
$\begingroup$

Say we have data where we have temperature, humidity, wind and rain for deciding whether we go and play tennis or not.

While selecting best feature to split upon initially, we use ID3 or Gini index. My question is why not use correlation ie when the features have maximum correlation with the output, split on that feature.

It seems obvious, but I can't find spit using correlation anywhere.

$\endgroup$
1
$\begingroup$

First of all, decision Trees can easily use categorical features with entropy and gini index, but (Pearson) correlation is defined for only numeric attributes. Using correlation with one-hot encoded versions of these categorical features also makes no sense. Secondly, In classification tasks, correlation between a, let's say binary output (e.g. $0,1$), and a numeric attribute may not make much sense as it normally does in regression problems. And, when it makes sense, it tries to capture the linear relationships by definition. (e.g. when $x=y^2$ and $x$ is symmetric for example, correlation is very low or zero)

For example, let's say we have a relationship $Y=1$ only when $X>0$, else $Y=0$. If we have small negative $X$, but comparably large positive $X$, as positive $X$'s move in positive/negative direction correlation changes because of their numeric values. But, actually this feature perfectly classifies the data.

While others are not perfect heuristics, correlation might be quite misleading when assessing the utility of the feature.

$\endgroup$
  • $\begingroup$ You just helped 20+ students. Thanks a lot. $\endgroup$ – Ashish Gupta Jan 12 at 15:53
1
$\begingroup$

As already pointed out by @gunes correlation would only be available if both the dependent variable and the regressor or splitting variable is numeric. But in that situation it certainly makes sense to consider correlation because it will perform well in capturing monotonic associations. Hence, the CTree algorithm Hothorn et al. (2006) available in the R package partykit (and its predecessor "party") uses this criterion by default for the numeric-vs-numeric situation (suitably standardized). See this discussion for more details: What is the test statistics used for a conditional inference regression tree?

The formulation of the splitting criteria in CTree is also such that it generalizes to other association tests when one or both of the variables are not numeric. For example, in the categorical-vs-categorical situation a $\chi^2$ statistic is used.

If it is a potential concern that associations are not monotonic, one potential idea is to use a maximally-selected statistic over all possible split points. This is also possible, see: Explanation of different testtype and teststats in ctree in party package of R

$\endgroup$
  • $\begingroup$ You just helped 20+ students. Thanks a lot. $\endgroup$ – Ashish Gupta Jan 12 at 15:53
  • $\begingroup$ Wow, glad I could help. $\endgroup$ – Achim Zeileis Jan 12 at 19:44

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.