I have 2 real and 1 discrete input variable whereas the output variable takes either of the 2 nominal values (i.e. 2 class problem). First I used Weka to train C 4.5 decision tree in a 10-fold cross validation setup. As a result I get a single decision tree whose True positive, False positive rates don't change much with the pruning factor.

What I want is a single variable whose value I can threshold (in addition to class identified by decision tree) to do classification. At the same time, I want readability of decision trees.

Is it possible to do this with regression trees ? Any starting points ?

Basically I want to know if it's possible to build a decision tree like in below image - enter image description here

  • 1
    $\begingroup$ I'm confused. If you have a single value (gotten somehow) then the tree will have only one branch. Also, this won't tell you much - it eliminates a lot of the benefit of trees. Or are you talking about two separate runs - one tree and one single number? $\endgroup$ – Peter Flom - Reinstate Monica Aug 30 '12 at 10:25
  • $\begingroup$ Thanks for the comment. What I want is a probability / some other measure of belongingness to a particular class at each node of the tree. $\endgroup$ – user13107 Aug 30 '12 at 10:51
  • $\begingroup$ @PeterFlom I have added an image to question for clarification. $\endgroup$ – user13107 Aug 30 '12 at 11:09
  • 1
    $\begingroup$ Any tree procedure will give you the probability of being in a class at each node. If you want a tree method that is based on significance and such, you might look at party in R. $\endgroup$ – Peter Flom - Reinstate Monica Aug 30 '12 at 11:09
  • $\begingroup$ Thanks @Peter, I will look at the mentioned pointer. I think (correct me if I'm wrong) the probability in Weka is computed as fraction of instances classified correctly during N-fold cross validation when decision tree was being trained. If there are some other methods, which depend not merely on # instances, but are a function of input variables, it'd be more insightful. $\endgroup$ – user13107 Aug 30 '12 at 11:15

It sounds like you want to use a method such a LMT (logistic model trees) or perhaps Joao Gama 's functional trees. The former uses standard decision tree splitting criteria at the interior nodes and additive logistic regression functions at the leaves, and the tree is pruned using the CART pruning method. The latter also can have logistic regression functions at the leaves and also at the interior nodes (for oblique splits). Weka has implementations of both: weka.classifiers.trees.LMT and weka.classifiers.trees.FT.

  • $\begingroup$ Thanks @MarkHall. I tried LMT and it seems to do the job of providing class probabilities as a function of input variables. I will plot the class probabilities on a test data and see how it behaves. $\endgroup$ – user13107 Aug 31 '12 at 17:51

Additional to the procedures suggested by @Mark Hall there also is the work of Loh e.g. LOTUS and GUIDE as well as of Zeileis et al. MOB. As compared to functional trees and LMT, these algorithms come from the statistics literature and use ideas of statistical inference for tree induction. Their distinguishing features are that they are more or less "unbiased" in split variable selection ("biased" here means that if there is no real association between the response and the predictors, predictors with more possible split points have a higher probability to be selected for the split. This bias appears in algorithms like C4.5 and CART and derivatives like LMT and Gama's trees). Unbiasedness is desirable if the tree is to be interpreted, rather than just used as a machine for predictions. See this paper or this thesis for a comparison of some of the mentioned algorithms and what they can do or what they are good at.

Now, what to use? I'm not exactly sure what you mean by "$x$ is a numerical attribute which corresponds to a probability of belonging to a particular class at a particular node", but if you want to regress given relative frequencies or estimated probabilities in the nodes, you can use a MOB with a beta regression model in the nodes, if you have a binary/k-ary outcome to be regressed you can use either LMT (I would not recommend it due to experience), LOTUS or MOB with binary/multinomial logistic regression.

  • $\begingroup$ Thanks @Momo for the suggestions and references (esp. the one by Rusch & Zeileis)! I will have to see how they perform on my data. Sorry, the site doesn't seem to allow me to accept two answers. $\endgroup$ – user13107 Aug 31 '12 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.