3
$\begingroup$

I would like some help from you in a classification model that I am developing.

In summary, the problem is:

– Classification problem with binary outcome (0/1)
– The classifier is a Random Forest with 1000 trees.
– Features are numeric and categorical, the latter were One-Hot-Encoded. Performed no scaling of numeric features as RF do not need it.
– Feature selection: based on variance (removed all features with variance below a certain threshold); the result was ~ 350 columns. Many of them are dummy variables as a result of OHE. Initial nº of features ~ 1200.

– The dataset has 45137 samples, with unbalanced classes:
* 66% Positives / Class 1
* 34% Negatives / Class 0

– Set up a Random Forest Classifier with scikit-learn.
– Model was trained with stratified 10-fold Cross-Validation.
– Train and test set ratios are: test size = 30%, train size = 70%, of whole dataset, obtained with train_test_split, with a stratified splitting. The splitting is random, using always the same seed.
– Performed parameter tuning with grid search: nr of estimators = [500, 1000] and mtry/max_features = [log2, sqrt, 0.8, 0.5]. Best results were for n_estimators = 1000 and max_features = 0.8

– Results are perfect in train, with clear distinction of the 2 classes, as can be seen in the probabilities scores plot. The scores in blue are the negatives, and the scores in orange are the positives.
– But results in test set are always much worse, mainly in the minority class, and I can´t figure out why. I think it is an overfit problem… But where is the problem? Too many (irrelevant) features that are adding noise? Bad parameter tuning? Class imbalance that needs solving?
Currently I am trying to solve class imbalance using class_weight = ‘balanced_subsample’. Is it enough or should I assign weights to the classes, or balance the dataset?

train_set_results

test_set_results

$\endgroup$
  • $\begingroup$ what are your c-validation results? $\endgroup$ – gunes Mar 15 at 18:43
  • 3
    $\begingroup$ Two comments on your parameter tuning: first, for a random forest, more trees are always better. This should not be a tuning parameter. Second, tree depth matters (a lot). You don't mention it here, but, really, you should do some parameter tuning on that as well. With respect to "unbalanced classes", this is actually a very well-balanced dataset compared to most; that's not your issue. With respect to removing low-variance features - consider the statement "performed no scaling of features as RF do not need it". This implies the variance of the feature is irrelevant... $\endgroup$ – jbowman Mar 15 at 19:33
  • 1
    $\begingroup$ I disagree when you say tree depth matters a lot. Random forest usually grows fully grown trees as the over fitting concerns of a single tree are somewhat mitigated by bootstrapping and growing many trees on different features $\endgroup$ – astel Mar 15 at 20:05
  • 1
    $\begingroup$ Also, depending on which implementation you are using one-hot encoding of features is not always necessary. For instance randomForest in R does not require it $\endgroup$ – astel Mar 15 at 20:08
  • 2
    $\begingroup$ @astel - in our experience, there's a big flat region in the "tree depth" space where tree depth doesn't matter much, but regions above and below that where it does. Computationally, trees that are way too big cost more time to build, and you need to build more of them to average out the overfitting, which also costs more time. So I don't entirely agree with you, but I don't entirely disagree either. $\endgroup$ – jbowman Mar 15 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.