I have a data set where some binary features divide the sample space roughly in half, whereas other features are much less frequent and occur only for 0.0001 - 0.01 of the sample space. However, those rare features are much more precise and can approach the level of sufficient conditions for the corresponding class.
I'm fitting a decision tree on this data but cannot find settings that prevent over-fitting on the frequent features while taking into account the rare features. If I set a maximum depth, it is either too low in general (overlooking rare features) or high enough in some paths but too high in others (over-fitting there). Similar problems occur for setting a minimal number of samples in a leaf/split.
I want to stick to a decision tree because I want a white-box model. Ideally, the rare features which are nearly sufficient conditions for their corresponding class would appear in the top of the tree, while the broader features are used as a best-effort classification when there is no applicable precise feature. Thus the tree would look something like below.
One option would be to divide the set of features into a precise and an imprecise set, first fit one tree on the precise features and then fit more trees on the imprecise set for those leaves that exceed some impurity measure. But that seems tedious, is there not a tree splitter strategy or different impurity measure that can handle data like this?