# How can 1 more feature disrupt a Random Forest's confusion matrix?

I'm trying to predict a binary variable with both random forests and logistic regression. I've got unbalanced classes (approx 1.5% of Y=1), so i'm calling class_weight = "auto" on both RF and LR.

I have approx 600 features and 262,871 lines.

### Here is the code :

model_rf = RandomForestClassifier(n_estimators = 500,
max_depth = None,
bootstrap = True,
criterion = "entropy",
class_weight = "auto",
n_jobs=-1)

# -- Features importance
forest = model_rf.fit(X_train[:,:580], Y_train)
imp_list = list(zip(db.columns[1:], np.transpose(forest.feature_importances_)))
imp_list.sort(key=operator.itemgetter(1), reverse=True)
imp_rf = pd.DataFrame(imp_list, columns=['features', 'imp'])
imp_rf[:20].plot(kind='barh', x='features', color='darkgreen')


### Confusion matrix are not so bad for both RF and LR

Feature impotances tend to say that the 8 new features are good (way better than the rest).

### The confusion matrix becomes very bad

My first thought was 'the model is overfitting'. But before trying to tune the RF, I removed all the 8 new variables and replaced them with 1 simulated random feature uncorrelated with the rest of the dataset.

• The variable was of course not 'important' on the RF
• (WHAT?) the confusion matrix was bad again.

How can 1 simple variable qualify as not important on 500 trees can disrupt the whole model ?

And again, the logit was stable :

What do you guys think ? Thanks for lending me your neurons.

The default feature importance techniques in random forests are based on classification accuracy. You should use feature importances based on AUCs: An AUC-based permutation variable importance measure for random forests. If your use R you can find their method implemented in the package party.
• I just tuned my parameters using roc_auc as a metric and I obtained an AUC = 82,5. So my guess is that the default parameters were not optimised. Thank you. About the feature importances, I posted a question referring to your article here for a scikit-learn implementation. – gowithefloww Jul 8 '15 at 14:42