I would like some help from you in a classification model that I am developing.
In summary, the problem is:
– Classification problem with binary outcome (0/1)
– The classifier is a Random Forest with 1000 trees.
– Features are numeric and categorical, the latter were One-Hot-Encoded. Performed no scaling of numeric features as RF do not need it.
– Feature selection: based on variance (removed all features with variance below a certain threshold); the result was ~ 350 columns. Many of them are dummy variables as a result of OHE. Initial nº of features ~ 1200.
– The dataset has 45137 samples, with unbalanced classes:
* 66% Positives / Class 1
* 34% Negatives / Class 0
– Set up a Random Forest Classifier with scikit-learn.
– Model was trained with stratified 10-fold Cross-Validation.
– Train and test set ratios are: test size = 30%, train size = 70%, of whole dataset, obtained with train_test_split, with a stratified splitting. The splitting is random, using always the same seed.
– Performed parameter tuning with grid search: nr of estimators = [500, 1000] and mtry/max_features = [log2, sqrt, 0.8, 0.5]. Best results were for n_estimators = 1000 and max_features = 0.8
– Results are perfect in train, with clear distinction of the 2 classes, as can be seen in the probabilities scores plot. The scores in blue are the negatives, and the scores in orange are the positives.
– But results in test set are always much worse, mainly in the minority class, and I can´t figure out why. I think it is an overfit problem… But where is the problem? Too many (irrelevant) features that are adding noise? Bad parameter tuning? Class imbalance that needs solving?
Currently I am trying to solve class imbalance using class_weight = ‘balanced_subsample’. Is it enough or should I assign weights to the classes, or balance the dataset?