5
$\begingroup$

I have been recently come across a problem that entails using a decision tree that only uses one continuous variable to divide the predictor on multiple threshold, while some splits result in the same decision. The decision tree is as follows: enter image description here

So I have two questions:

  • Does it make sense to use one variable multiple times for dividing the predictor?
  • Is it possible that two splits result in the same decision?

I didn't share any codes or data set because I just want to know whether this condition is possible and why we would prefer this.

$\endgroup$

3 Answers 3

3
$\begingroup$

Does it make sense to use one variable multiple times for dividing the predictor?

Yes, if the variable is useful and has different regions that are not connnected, it can be used mutliple times, e.g. return 1 if 1<x<4 else 0 would require two splits.

Is it possible that two splits result in the same decision?

It is possible as well. Although the final decision is the same, which might have been obtained by taking the majority of cases in each branch, the overall entropy might improve with the branching.

For example, if the distribution is 5-7 for class 0 and 1 respectively, and after the split it becomes (2-3) and (3-4) in the branches, the information gain would be:

h  = -(log(5/12) * (5/12) + log(7/12) * (7/12)) 
b1 = -(log(2/5) * (2/5) + log(3/5) * 3/5)
b2 = -(log(3/7) * (3/7) + log(4/7) * 4/7)
h - (b1 * 5/12 + b2 * 7/12)

which would result in the following gain

0.000408677
$\endgroup$
5
  • $\begingroup$ Thank you very much for your thorough explanation. So do you think pruning also could be a choice here as from a stage on the dividing process won't yield a new result. $\endgroup$ Dec 16, 2021 at 22:04
  • 1
    $\begingroup$ Yes, certainly. $\endgroup$
    – gunes
    Dec 16, 2021 at 22:14
  • $\begingroup$ Could you please explain the cross entropy you used here? I would be very grateful. $\endgroup$ Dec 17, 2021 at 12:45
  • 1
    $\begingroup$ Hi @AnoushiravanR it's the usual one, the entropy: $H(p)=\sum p \log p$. Calculate it for each branch, weighted sum them and subtract it from the entropy calculated in the root. That is the information gain. $\endgroup$
    – gunes
    Dec 17, 2021 at 13:32
  • 1
    $\begingroup$ Thank you very much for your comment. Yes I should have noticed. Normally we used Gini index, so this was quite new to me, but very glad that I came to know it now. $\endgroup$ Dec 17, 2021 at 14:20
4
$\begingroup$
  1. Absolutely. In one particular case, you may have a non-monotonic relationship between the continuous variable and the target, so that the "correct" tree will have several switches from No to Yes along the leaves from left to right.

  2. Yes. While the final "answer" Yes/No can be the same in two leaves from the same parent, their proportions of "yes" samples might be quite different, which may be informative for you. If at the end of the day you only care about the final decision, you can always post-process the tree to simplify the rules.

$\endgroup$
1
  • $\begingroup$ Thank you for your answers dear Ben. So do you think pruning here could be a good choice cause from a stage below dividing further does not make any difference. $\endgroup$ Dec 16, 2021 at 21:53
3
$\begingroup$

YES & YES

set.seed(2021)
N <- 10
x0 <- c(runif(N, -4, -2), runif(N, 2, 4))
x1 <- runif(2*N, -2, 2)
plot(x0, rep(1, 2*N), col = 'red')
points(x1, rep(1, 2*N), col = 'blue')

(If someone could run this in R and post the picture, that would be fantastic.)

If you're above $0$, the color depends on if you're above $2$. If you're below $0$, the color depends on if you're below $-2$. However, you cannot split at $2$ or $-2$ and always get the right color. There are reds and blues above $-2$. There are reds and blues below $2$. Splitting on the same variable multiple times allows us to guess the correct color every time.

In some pseudocode:

if x > 0:
    if x < 2:
        color = red
    if x > 2:
        color = blue
if x < 0:
    if x < -2:
        color = red
    if x > -2:
        color = blue
$\endgroup$
2
  • $\begingroup$ Thank you very much for your code dear Dave. I ran it but I am not sure how I interpret it. $\endgroup$ Dec 16, 2021 at 21:54
  • 1
    $\begingroup$ You can add it via rdrr.io/snippets $\endgroup$
    – gunes
    Dec 16, 2021 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.