I am using a decision tree to perform binary classification. I've found that a particular feature seems to be an important one; however, keeping it in my model doesn't yield better predictions (i.e. results in a near equivalent confusion matrix as that without the feature). Does this just mean that the feature is highly correlated with another/other feature(s) already in the model?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ How do you know that it's important if it doesn't improve the model? Looking at a confusion matrix is deceptive because it only looks at $\hat{y}>c$ rather than the model's confidence in the prediction, cf. proper vs. improper scoring rules. $\endgroup$– Sycorax ♦Mar 21, 2016 at 21:32
-
$\begingroup$ I have the feature importances relative to the other features used in the model. I don't have the precise calculation, but I think it demonstrates that features nearer/at the top of the tree contribute to the prediction of a larger fraction of the input. $\endgroup$– dnbwiseMar 21, 2016 at 23:34
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Yes, that seems to be the likely reason why you're not seeing an improvement in the model performance as you add that feature in your model.
In order to confirm this, you can take all features that are in the model, add the feature of interest, and perform a multicollinearity diagnostic (e.g., VIF) on that set of features. (As a first step though, you could just review pairwise correlations among these features to check if there's another single feature that is highly correlated with the feature of interest.)
Also, based on the tags in your question, it looks like you're using CART -- which splits each parent node into only two child nodes at a time. You may need to grow the tree further by splitting the tree multiple times with that same feature of interest. (Make sure that tree criteria -- like split size, leaf node size, and depth are not restricting the growth of the tree.) Alternatively, you can try other decision tree algorithms (e.g., CHAID) that go beyond binary splits.