5
$\begingroup$

I'm trying to predict a binary variable with both random forests and logistic regression. I've got unbalanced classes (approx 1.5% of Y=1), so i'm calling class_weight = "auto" on both RF and LR.

I have approx 600 features and 262,871 lines.

Here is the code :

model_rf = RandomForestClassifier(n_estimators = 500,
                                  max_depth = None,
                                  bootstrap = True,
                                  criterion = "entropy",
                                  class_weight = "auto",
                                  n_jobs=-1)

# -- Features importance
forest = model_rf.fit(X_train[:,:580], Y_train)
imp_list = list(zip(db.columns[1:], np.transpose(forest.feature_importances_)))
imp_list.sort(key=operator.itemgetter(1), reverse=True)
imp_rf = pd.DataFrame(imp_list, columns=['features', 'imp'])
imp_rf[:20].plot(kind='barh', x='features', color='darkgreen')

feature importance before adding new features

Confusion matrix are not so bad for both RF and LR

CM without adding

After adding 8 features

Feature impotances tend to say that the 8 new features are good (way better than the rest). enter image description here

The confusion matrix becomes very bad

enter image description here

Adding 1 simulated feature

My first thought was 'the model is overfitting'. But before trying to tune the RF, I removed all the 8 new variables and replaced them with 1 simulated random feature uncorrelated with the rest of the dataset.

  • The variable was of course not 'important' on the RF
  • (WHAT?) the confusion matrix was bad again.

How can 1 simple variable qualify as not important on 500 trees can disrupt the whole model ?

enter image description here

And again, the logit was stable :

enter image description here

What do you guys think ? Thanks for lending me your neurons.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Classification accuracy is not a good metric when your dataset is unbalanced. You should use AUC.

The default feature importance techniques in random forests are based on classification accuracy. You should use feature importances based on AUCs: An AUC-based permutation variable importance measure for random forests. If your use R you can find their method implemented in the package party.

Improve this answer
$\endgroup$
4
  • $\begingroup$ You right. AUC is the metric I use to compare my models indeed. Features importance plot was just to add some information to my post. Moreover if I follow you correctly : adding 8 features which enhanced classification accuracy (in Y=0) implies that they are more often selected in each split of the RF and so, disrupt the AUC. But then, what's happening with the add of 1 simulated feature - which is clearly not boosting classification accuracy ? $\endgroup$
    – gowithefloww
    Jul 8, 2015 at 8:29
  • 1
    $\begingroup$ I guess that classification accuracy is not a good metric after all. Try to repeat everything with AUC. Look up for Prof. Frank Harrell answers about classification accuracy: he often showed to be strongly against its use. $\endgroup$
    – Simone
    Jul 8, 2015 at 9:24
  • 1
    $\begingroup$ I just tuned my parameters using roc_auc as a metric and I obtained an AUC = 82,5. So my guess is that the default parameters were not optimised. Thank you. About the feature importances, I posted a question referring to your article here for a scikit-learn implementation. $\endgroup$
    – gowithefloww
    Jul 8, 2015 at 14:42
  • $\begingroup$ Just a brief comment. The accuracy associated to the random forest confusion matrix are respectively: 0.78, 0.97, and 0.99. However, you notice the last two confusion matrices look worse than the first. Feature importance based on "decrease in accuracy" might be unreliable here just because accuracy is unreliable. $\endgroup$
    – Simone
    Jul 9, 2015 at 0:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.