Samples in decision trees

Question

I am trying to do some binary decision trees with Python (scikit-learn), but my sample has a bad repartition : I have something like 100 000 data points with label 0 and 800 000 with the label 1.

So when I get the tree, I don't have a lot of leaves with class 0, and I don't even have any before reaching a depth of 5. And in that node, there are very few points.

I also tried trees with other variables (still the same labels) and I get more data with label 0 ; the problem is that I am not sure this makes sense.

What could I do ? Should I take a sample of my data with 50% of each label? I am afraid doing that won't really be representative of my data, especially if I put new data in the tree afterwhat. Does anyone know what are the requirements for samples for decision trees? I didn't find any information anywhere about it.

Answer 1

You can solve this problem by going for Stratified K Fold cross validation.

K-Folds cross-validator provides train/test indices to split data in train/test sets. Split dataset into k consecutive folds (without shuffling by default). Each fold is then used once as a validation while the k - 1 remaining folds form the training set.

Stratified K-Folds cross-validator provides train/test indices to split data in train/test sets.This cross-validation object is a variation of KFold that returns stratified folds. The folds are made by preserving the percentage of samples for each class.

You can refer here for the documentation and examples.

Answer 2

I don't see why cross-validation (see answer by @phanny) would make things any better in this case. What you may want to look at is an oversampling/undersampling technique. Generally, "plain old oversampling" the minority class leads to overfitting, so I would recommend to either go for SMOTE https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume16/chawla02a-html/chawla2002.html or for combining undersampling and cross-validation (perform each round of cross-validation training with an equal number of cases from each label, so, basically, what you suggested in the final paragraph of your question, but with cross-validation involved) For more info, check https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis

Anyway, don't expect any miracles! Unbalanced data is one of the major problems in classification problem, specially in "harder" ones (in the sense of there not being a simple separation between the classes) This compensating techniques always risk to overestimate the chance of any individual being in the minority class, which is OK (compared to the opposite case) in some problems like medical diagnose, but may not be in your particular problem.

Finally, consider usgin a Random Forest (or AdaBoost) to increase the performance of your trees. Also, don't use overall accuracy as a validation metric for problems with unbalanced data. Try a precision-recall compromise instead.

Answer 3

100k samples per class is a lot, even for quite high dimensional inputs. So I suggest undersampling and use 100k of each class. The samples must be drawn randomly to avoid introducing bias.

To control the tree depths (and the tendency to overfit), use min_samples_leaf. It decides the minimum samples in a single leaf node. To find the appropriate setting for this hyperparameter, use cross validation. Maybe try values in the range 1-1000, log-spaced.

Using a RandomForest instead of a single DecisionTree will greatly reduce the risk of overfitting.

To see how well your model generalized to new data you must use a test set.