6
$\begingroup$

Assume I'm a doctor and I want to know which variables are most important to predict breast cancer (binary classification). Two different scientists each present me with a different feature importance figure...

Logistic Regression with L2 norm (absolute values of model coefficients; 10 highest shown): enter image description here

And Random Forests (10 highest shown): enter image description here

The results are very different. Which scientist should I trust? Are one/both of these figures meaningless?

Code below; using the Wisconsin Breast Cancer data-set in scikit-learn.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegressionCV
from sklearn.ensemble import RandomForestClassifier
import numpy as np
import matplotlib.pyplot as plt

data = load_breast_cancer()
y = data.target
X = data.data

clf = LogisticRegressionCV(max_iter=3000)
clf.fit(X, y)
coefs = np.abs(clf.coef_[0])
indices = np.argsort(coefs)[::-1]

plt.figure()
plt.title("Feature importances (Logistic Regression)")
plt.bar(range(10), coefs[indices[:10]],
       color="r", align="center")
plt.xticks(range(10), data.feature_names[indices[:10]], rotation=45, ha='right')
plt.subplots_adjust(bottom=0.3)

clf = RandomForestClassifier(n_jobs=-1, random_state=42, n_estimators=400, max_depth=6, max_features=6) #has already been tuned
clf.fit(X, y)
coefs = clf.feature_importances_
indices = np.argsort(coefs)[::-1]

plt.figure()
plt.title("Feature importances (Random Forests)")
plt.bar(range(10), coefs[indices[:10]],
       color="r", align="center")
plt.xticks(range(10), data.feature_names[indices[:10]], rotation=45, ha='right')
plt.subplots_adjust(bottom=0.3)

plt.ion(); plt.show()
|cite|improve this question
$\endgroup$
  • 5
    $\begingroup$ I don't know Python that well, but are you using the coefficient values to assess importance for logistic regression? If so, you need to account for the standard errors. A meaningless variable may have a large coefficient, but also a large standard error. Both models are also affected by multicollinearity. You should see how removing a few variables affect your final importance rankings. $\endgroup$ – Peter Calhoun Mar 24 '18 at 2:53
  • 5
    $\begingroup$ It's not that they are meaningless, it's that they measure something about the model not something about the way the world works. There is no agreed upon notion of the predictive power of a variable, it's a completely informal and nebulous concept. It's only ever defined in the context of a specific model, so it ends up telling you more about the model than the world. $\endgroup$ – Matthew Drury Mar 24 '18 at 3:00
  • 6
    $\begingroup$ There's not a single definition of "importance" and what is "important" between LR and RF is not comparable or even remotely similar; one RF importance measure is mean information gain, while the LR coefficient size is the average effect of a 1-unit change in a linear model. Gary King describes in that article why even standardized units of a regression model are not so simply interpreted. Moreover, the LR and RF feature importance concepts are so far removed from each-other that grouping them under "feature importance" is really, really deceptive. $\endgroup$ – Sycorax says Reinstate Monica Mar 24 '18 at 3:13
  • 2
    $\begingroup$ The idea that one measure is "right" completely misses the point that LR and RF provide completely different answers to the same question by their very nature, and that is doubly true when seeking to know what is "important." $\endgroup$ – Sycorax says Reinstate Monica Mar 24 '18 at 3:16
  • 2
    $\begingroup$ @OliverAngelil Why would you want a doctor to make a decision that way? The model is built to use all of that information in conjunction. $\endgroup$ – Sycorax says Reinstate Monica Mar 24 '18 at 5:54
1
$\begingroup$

From your comments, it seems like what you are really after is feature selection - you want a set of models that use variable numbers of features (1, 2, 3, ..., N), such that incrementally adding a new feature yields as great an increase in model performance as possible. Then the decision makers can assess whether they want to carry out a costly procedure to obtain the data for an additional feature to use a more complicated model with greater precision/recall. We assume here that it costs the same to obtain the data for each feature.

In that case, I would separate your data into a training and test set; I would use cross-validation on the training set to select the best incremental feature (strictly speaking, you need to use nested cross-validation here, but if that is computationally infeasible or you don't have enough data we can verify that we did not overfit by cross-referencing CV results with test set results at the end). That is, you would start by trying each feature on their own, and choose the feature that gives you the best CV performance. You would then repeat the process to iteratively add additional features.

Note whether different CV folds show up with different best incremental features - if the variability is too high, this approach may not be feasible.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Don't you think what features are picked next to improve the model most will depend on the ML method used? E.g. logistic regression vs random forest. Or is the method irrelevant, but rather whatever one leads to the biggest improvement in test error? $\endgroup$ – Oliver Angelil Apr 5 '18 at 16:36
  • 2
    $\begingroup$ @OliverAngelil Yes, it might depend on the model used. So it makes sense to perform such feature selection on the model that you will actually be using, e.g. the one with the best out-of-sample performance. You might even want to ensemble several models, it doesn't matter - you perform this kind of feature selection using the model that you end up using. Different models giving you different important features is not necessarily a problem - it might indicate high variance, or maybe multicollinearity, or maybe your two models have low correlation in which case you should ensemble them. $\endgroup$ – rinspy Apr 6 '18 at 8:22
  • 2
    $\begingroup$ @OliverAngelil Of those cases, I would say only high variance is a problem for a predictive model. If you aim to establish some causality relationship to infer some knowledge from your model, it's a different story, of course. $\endgroup$ – rinspy Apr 6 '18 at 8:23
0
$\begingroup$

The question is ill-posed. We cannot advise the doctor that, for example, inspecting feature $X_a$ is more worthwhile than inspecting feature $X_b$, since how "important" a feature is only makes sense in the context of a specific model being used, and not the real world.

Logistic Regression and Random Forests are two completely different methods that make use of the features (in conjunction) differently to maximise predictive power. This is why a different set of features offer the most predictive power for each model.

Such feature importance figures often show up, but the information they are thought to convey is generally mistaken to be relevant to the real world. Why one would be interested in such a feature importance is figure is unclear.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.