I am doing binary classification with decision tree, and it aims to find out what features matter the most with the data we have, so I need interpretability more than predictability. It is like feature selection with CART.
The problem is, the dataset is highly unbalanced (True/False is about 1/100). I found some discussions about how to handle such situation when training predictive model, including oversampling or undersampling to increase the accuracy. However, for explanatory models without training and testing set, accuracy doesn't seem like a good index.
I am considering three different methods to build my decision tree, and hoping you to help me know which one should be the best and why it is.
Simply use all the data in the tree without any adjustment.
I believe that this can be the worst option since the tree will be highly biased to the "False" group, and I am getting nothing interpretable because every nodes would be "False"-dominant. Still I'd like to hear second opinions.
Get a random sample from the "False" group with the same size of the "True" group.
Assuming that the sample is representative of the "False" population, I believe this method can effectively figure out the key differences between the two groups without worrying the imbalance. The problem is how "representative" the sample actually is. Should I repeat the process numerous times to make sure of that?
Use all the data but adjust
With my understanding from how does the class_weight parameter in scikit-learn work, that parameter is designed to solve the unbalanced-dataset problem. Without anything to presume, I set that weight also as 1/100, and I am getting a similar tree like that built from the previous method with sampling. I wonder whether this method is better.
Any thoughts or suggestions are very appreciated.