I have a data set where some binary features divide the sample space roughly in half, whereas other features are much less frequent and occur only for 0.0001 - 0.01 of the sample space. However, those rare features are much more precise and can approach the level of sufficient conditions for the corresponding class.

I'm fitting a decision tree on this data but cannot find settings that prevent over-fitting on the frequent features while taking into account the rare features. If I set a maximum depth, it is either too low in general (overlooking rare features) or high enough in some paths but too high in others (over-fitting there). Similar problems occur for setting a minimal number of samples in a leaf/split.

I want to stick to a decision tree because I want a white-box model. Ideally, the rare features which are nearly sufficient conditions for their corresponding class would appear in the top of the tree, while the broader features are used as a best-effort classification when there is no applicable precise feature. Thus the tree would look something like below.

One option would be to divide the set of features into a precise and an imprecise set, first fit one tree on the precise features and then fit more trees on the imprecise set for those leaves that exceed some impurity measure. But that seems tedious, is there not a tree splitter strategy or different impurity measure that can handle data like this?

enter image description here

  • $\begingroup$ You could define your own impurity based on the max precision of the split nodes. if you are using a package then it depends on whether your package allows you to define custom impurities. $\endgroup$ – knk Dec 3 '17 at 21:35
  • $\begingroup$ However, more importantly think of the complexity you are generating by splitting on the "rare" features. To arrive at decision on average you are running through the whole tree, which is bad. $\endgroup$ – knk Dec 3 '17 at 21:52
  • $\begingroup$ @knk thanks for your thoughts. I'm using sklearn in Python, which doesn't normally allow other criteria besides gini and entropy, but I'm trying to hack it in. Yes, you are right classification will be slower, but in my case that is not an issue; I'm only using the classifier to get more insight in the data, not to make live decisions or so, and the dataset is relatively small. $\endgroup$ – user98500 Dec 3 '17 at 22:25
  • $\begingroup$ @knk I got it working, but then broad features are being used far down the tree in niches where they appear to be discriminative (actually due to spurious correlations). I'm now thinking of taking my original approach, first using only rare features with some high entropy threshold and then a best-effort on the remaining data with the coarser features. Or do you have other ideas? $\endgroup$ – user98500 Dec 7 '17 at 11:53
  • $\begingroup$ It looks like this is a particular decision scenario for deeper data analysis, in that case I would also ask you look at projection techniques like Principal component analysis and then choose the decision nodes based on the eigenvalues and the entropy. I have no idea how to combine them because that would depend on your needs. $\endgroup$ – knk Dec 7 '17 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy