I'm trying to predict a binary variable with both random forests and logistic regression. I've got unbalanced classes (approx 1.5% of Y=1), so i'm calling class_weight = "auto" on both RF and LR.

I have approx 600 features and 262,871 lines.

Here is the code :

model_rf = RandomForestClassifier(n_estimators = 500,
                                  max_depth = None,
                                  bootstrap = True,
                                  criterion = "entropy",
                                  class_weight = "auto",
                                  n_jobs=-1)

# -- Features importance
forest = model_rf.fit(X_train[:,:580], Y_train)
imp_list = list(zip(db.columns[1:], np.transpose(forest.feature_importances_)))
imp_list.sort(key=operator.itemgetter(1), reverse=True)
imp_rf = pd.DataFrame(imp_list, columns=['features', 'imp'])
imp_rf[:20].plot(kind='barh', x='features', color='darkgreen')

feature importance before adding new features

Confusion matrix are not so bad for both RF and LR

CM without adding

After adding 8 features

Feature impotances tend to say that the 8 new features are good (way better than the rest). enter image description here

The confusion matrix becomes very bad

enter image description here

Adding 1 simulated feature

My first thought was 'the model is overfitting'. But before trying to tune the RF, I removed all the 8 new variables and replaced them with 1 simulated random feature uncorrelated with the rest of the dataset.

  • The variable was of course not 'important' on the RF
  • (WHAT?) the confusion matrix was bad again.

How can 1 simple variable qualify as not important on 500 trees can disrupt the whole model ?

enter image description here

And again, the logit was stable :

enter image description here

What do you guys think ? Thanks for lending me your neurons.

up vote 1 down vote accepted

Classification accuracy is not a good metric when your dataset is unbalanced. You should use AUC.

The default feature importance techniques in random forests are based on classification accuracy. You should use feature importances based on AUCs: An AUC-based permutation variable importance measure for random forests. If your use R you can find their method implemented in the package party.

  • You right. AUC is the metric I use to compare my models indeed. Features importance plot was just to add some information to my post. Moreover if I follow you correctly : adding 8 features which enhanced classification accuracy (in Y=0) implies that they are more often selected in each split of the RF and so, disrupt the AUC. But then, what's happening with the add of 1 simulated feature - which is clearly not boosting classification accuracy ? – gowithefloww Jul 8 '15 at 8:29
  • 1
    I guess that classification accuracy is not a good metric after all. Try to repeat everything with AUC. Look up for Prof. Frank Harrell answers about classification accuracy: he often showed to be strongly against its use. – Simone Jul 8 '15 at 9:24
  • 1
    I just tuned my parameters using roc_auc as a metric and I obtained an AUC = 82,5. So my guess is that the default parameters were not optimised. Thank you. About the feature importances, I posted a question referring to your article here for a scikit-learn implementation. – gowithefloww Jul 8 '15 at 14:42
  • Just a brief comment. The accuracy associated to the random forest confusion matrix are respectively: 0.78, 0.97, and 0.99. However, you notice the last two confusion matrices look worse than the first. Feature importance based on "decrease in accuracy" might be unreliable here just because accuracy is unreliable. – Simone Jul 9 '15 at 0:10

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.