I have features
C. As an exercise in feature selection and feature engineering, I'd like to see what information I can get from two tests:
1) Seeing how well each feature performs, ALONE, in a classifier.
2) Observing the feature importances of each feature, when using all features in the same classifier.
Let's consider two scenarios on two DIFFERENT datasets after running 1) and 2), above:
1) AUC of features used alone 2)Feature importance in total model (AUC=0.7) A 0.55 0.37 B 0.57 0.09 C 0.60 0.54
Since the features all have similar performances, but their feature importances are wildly different, it makes me think that there is some multicollinearity, and strong interactions between features.
scenario 2 (on a different dataset)
1) AUC of features used alone 2) Feature importance in total model (AUC=0.7) A 0.13 0.37 B 0.75 0.29 C 0.61 0.34
In this scenario, my hypothesis is that the features are uncorrelated with each other, and that the interactions between variables create more powerful predictors.
Is either or both of my hypotheses correct for scenarios
I'm using a Random Forest Classifier for binary classification, so the performance metric is
AUC, and the feature importances are
gini importance. but the principles should apply to other classifiers in general, too.
Part of my assumption here, is that similar performance of two features means that they are correlated. In the case of RandomForest, I can't wrap my head around whether or not this is true, since the performance is computed by
AUC, and there are many ways to get the same