8
$\begingroup$

In Applied Predictive Modeling by Kuhn and Johnson the authors write:

Finally, these trees suffer from selection bias: predictors with a higher number of distinct values are favored over more granular predictors (Loh and Shih,   1997;   Carolin et   al.,   2007;   Loh,   2010). Loh and Shih  ( 1997) remarked that “The danger occurs when a data set consists of a mix of informative and noise variables, and the noise variables have many more splits than the informative variables. Then there is a high probability that the noise variables will be chosen to split the top nodes of the tree. Pruning will produce either a tree with misleading structure or no tree at all.”

Kuhn, Max; Johnson, Kjell (2013-05-17). Applied Predictive Modeling (Kindle Locations 5241-5247). Springer New York. Kindle Edition.

They go on to describe some research into building unbiased trees. For example Loh's GUIDE model.

Staying as strictly as possible within the CART framework, I'm wondering if there's anything I can do to minimize this selection bias? For example, perhaps clustering/grouping high cardinality predictors is one strategy. But to what degree should one do the grouping? If I have a predictor with 30 levels should I group to 10 levels? 15? 5?

Improve this question
$\endgroup$
7
  • $\begingroup$ Here is a related question and answer. $\endgroup$
    – dal233
    Commented Dec 31, 2015 at 19:14
  • 1
    $\begingroup$ Keep in mind that CART is not just biased against factors with many levels, but potentially continuous variables too if your sample size is large. Is there a particular reason you want to stay within the CART framework? In addition to GUIDE, conditional inference trees are another option to avoid selection bias. $\endgroup$
    – dmartin
    Commented Jan 6, 2016 at 1:45
  • $\begingroup$ My impression is that there is more off-the-shelf code written for CART and in addition, I want to keep things simple to explain. $\endgroup$
    – dal233
    Commented Jan 6, 2016 at 19:13
  • $\begingroup$ When I said "off the shelf code written for CART" - I also meant the whole ecosystem around CART. Like for example rpart.plot. $\endgroup$
    – dal233
    Commented Jan 7, 2016 at 22:02
  • $\begingroup$ ?ctree and you'll see the party package has many of the same features that rpart does. Missing data is handled via surrogate splits as well $\endgroup$
    – dmartin
    Commented Jan 7, 2016 at 22:29

1 Answer 1

Reset to default
2
$\begingroup$

Based on your comment I'd go with a conditional inference framework. The code is readily available in R using the ctree function in the party package. It has unbiased variable selection, and while the algorithm underlying when and how to make splits is different compared to CART, the logic is essentially the same. Another benefit outlined by the authors (see the paper here) is that you don't have to worry so much about pruning the tree to avoid overfitting. The algorithm actually takes care of that by using permutation tests to determine whether a split is "statistically significant" or not.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.