4
$\begingroup$

I was wondering if someone can explain to me why in my decision tree some of the variable with the highest importance (highest information - script shown below) do not appear into my tree at all, while the one in the first node (which from my understanding is the one doing the best split) is ranked 6th in terms of information gain!

I had run the decision tree using the rpart library in R and then pruned the tree using the following code:

tree <- rpart(factor(PREDICTOR) ~ ...
          ,data = data)

printcp(tree) # display the results

--bestcp*root node errror

bestcp <- tree$cptable[which.min(tree$cptable[,"xerror"]),"CP"]
tree.pruned <- prune(tree, cp = bestcp) 

Afterwards I run the function published by looking at that post:How to measure/rank "variable importance" when using CART? (specifically using {rpart} from R). I also run the same process using SPSS with the CRT method. It gave me an almost identical tree as rpart and an identical variable importance table.

$\endgroup$
3
$\begingroup$

Based on your statement I believe that your data set must be having Super Attributes hence Gain Ratio would have been considered instead of Information Gain for splitting.

Super Attributes are predictors which have more number of factor levels and Information Gain is biased towards attributes having more factor levels than attributes having less factor levels.

$\endgroup$
  • $\begingroup$ I see. I think you are right. I search it a little bit and it seems that it is as you said. rpart uses the Gini index to make the split and not the information gain and it is also true that the variables excluded from the tree are continuous, whch adds up to your conclusion.Thanks! $\endgroup$ – ALEX.VAMVAS Jul 25 '17 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.