I'm trying to predict a binary variable with both random forests and logistic regression. I've got unbalanced classes (approx 1.5% of Y=1), so i'm calling
class_weight = "auto" on both RF and LR.
I have approx 600 features and 262,871 lines.
Here is the code :
model_rf = RandomForestClassifier(n_estimators = 500, max_depth = None, bootstrap = True, criterion = "entropy", class_weight = "auto", n_jobs=-1) # -- Features importance forest = model_rf.fit(X_train[:,:580], Y_train) imp_list = list(zip(db.columns[1:], np.transpose(forest.feature_importances_))) imp_list.sort(key=operator.itemgetter(1), reverse=True) imp_rf = pd.DataFrame(imp_list, columns=['features', 'imp']) imp_rf[:20].plot(kind='barh', x='features', color='darkgreen')
Confusion matrix are not so bad for both RF and LR
After adding 8 features
Feature impotances tend to say that the 8 new features are good (way better than the rest).
The confusion matrix becomes very bad
Adding 1 simulated feature
My first thought was 'the model is overfitting'. But before trying to tune the RF, I removed all the 8 new variables and replaced them with 1 simulated random feature uncorrelated with the rest of the dataset.
- The variable was of course not 'important' on the RF
- (WHAT?) the confusion matrix was bad again.
How can 1 simple variable qualify as not important on 500 trees can disrupt the whole model ?
And again, the logit was stable :
What do you guys think ? Thanks for lending me your neurons.