I have features A, B and C. As an exercise in feature selection and feature engineering, I'd like to see what information I can get from two tests:

1) Seeing how well each feature performs, ALONE, in a classifier.

2) Observing the feature importances of each feature, when using all features in the same classifier.

Let's consider two scenarios on two DIFFERENT datasets after running 1) and 2), above:

scenario 1

 1) AUC of features used alone      2)Feature importance in total model (AUC=0.7)

A  0.55                              0.37
B  0.57                              0.09
C  0.60                              0.54

Since the features all have similar performances, but their feature importances are wildly different, it makes me think that there is some multicollinearity, and strong interactions between features.

scenario 2 (on a different dataset)

   1) AUC of features used alone      2) Feature importance in total model (AUC=0.7)

    A    0.13                         0.37
    B    0.75                         0.29
    C    0.61                         0.34

In this scenario, my hypothesis is that the features are uncorrelated with each other, and that the interactions between variables create more powerful predictors.

Is either or both of my hypotheses correct for scenarios 1 and 2?

I'm using a Random Forest Classifier for binary classification, so the performance metric is AUC, and the feature importances are gini importance. but the principles should apply to other classifiers in general, too.

Part of my assumption here, is that similar performance of two features means that they are correlated. In the case of RandomForest, I can't wrap my head around whether or not this is true, since the performance is computed by AUC, and there are many ways to get the same AUC.

  • 1
    $\begingroup$ What are the AUC's in scenario 2? You describe scenario 2 as "using all features in the same classifier.". How then can each feature have a different AUC? $\endgroup$ – Matthew Drury Jan 17 '16 at 2:25
  • 1
    $\begingroup$ It is also not true that having a similar importance is at all related to correlation. For example, if you sample $X_1, X_2 \in U(-1, 1)$ independently, and let the response be the "inside the circle" indicator $Y = (X_1^2 + X_2^2 < 1)$, then in a random forest model $Y \sim X_1 + X_2$, the features $X_1, X_2$ will have the same importance, even though they are independent. $\endgroup$ – Matthew Drury Jan 17 '16 at 2:27
  • $\begingroup$ That is useful information, but I propose that features of similar performance (not importance, see scenario 1) are related to correlation. $\endgroup$ – Candic3 Jan 17 '16 at 2:49
  • $\begingroup$ @Candic3 Wouldn't $X_1$ and $X_2$ have the same performance in Matt's example? Also you may want to edit or take out the first table because it's confusing given what you're trying to show. $\endgroup$ – dsaxton Jan 17 '16 at 3:28
  • $\begingroup$ @MatthewDrury, the "two tests" that I refer to do not mean scenario 1 and scenario 2, but rather they mean 1) AUC of features used alone and 2)Feature importance in total model (AUC=0.7) in each of the scenarios. The scenarios are conducted on two different datasets, but the two tests are done on each dataset $\endgroup$ – Candic3 Jan 17 '16 at 3:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.