I am using logistic regression for binary classification. I have a big data set (happens to be highly unbalanced: 19 : 1). So I use scikit-learn's LogisticRegression() to train on 80% of my labelled data and then I validated with the other 20% (I looked at area under ROC as well as precision-recall because the data was so unbalanced; I also used the model with class_weight='auto').

My main question is the following: once I start generating predictions of unlabelled input vectors (using predict_proba()), how can I tell which of the features contributed the most to the prediction of that particular input? I imagine that this might be different than the "most important features" as determined in general for the model based on the labelled training data (e.g., coefficient magnitude).

I had a very basic idea:

  1. Take the component-wise product of my input feature values with the absolute value of my feature coefficients. The most contributing feature is then the one that corresponds to the entry with the largest value.

  2. Do (1) but use z-scores for everything (training and input features). I thought this would be important because I worried that some feature ranges might be very different from others and just taking products might not capture this; but I guess the coefficients should reflect ranges so maybe this doesn't matter.

Any thoughts would be greatly appreciated as I'm new to this. Things specific to logistic regression (i.e., sigmoid instead of just linear function) and any references to how to implement specific actions (e.g., transforms) in scikit-learn would be greatly appreciated as I actually am doing a project with real data.

  • $\begingroup$ Isn't logistic regression much of a predictive model rather than an explanatory type? $\endgroup$
    – tagoma
    Aug 2 '17 at 22:17
  • $\begingroup$ @tagoma it's both, right? $\endgroup$
    – Firebug
    Nov 7 '17 at 20:48

There is a way using the regression coefficients only, you can understand which features most contribute to the prediction of a given input vector.

However you will have to standardize and scale each variable first (i.e. subtract the mean and divide by the standard deviation). Then refitting your model with the standardized & scaled data, the feature with the largest regression coefficient will be the feature that contributes the most to future predictions.

The regression coefficients are comparable after scaling because we have made the units of the features irrelevant, thus a one unit increase in feature $X_1$ corresponds to jumping 1 standard deviation of the unscaled feature.

  • $\begingroup$ Alejandro, thanks for your answer. There is one issue with training on normalized data. I obtain far worse model performance. My area under the ROC curve is about 10% less and my area under the precision-recall curve is also worse. Thus, I am hesitant to switch my model fitting to normalized data. Is this the cost of getting the individual feature importance I am looking for? Is there another way? Doesnt the magnitude of the coefficients reflect their importance? $\endgroup$ Jun 25 '15 at 6:56
  • $\begingroup$ Hmm thats very strange. I wouldn't expect normalizing the data to effect your estimates since normalizing does not effect the underlying relationships between variables. I'm not sure of any other way for comparing predictive value of individual features $\endgroup$ Jun 27 '15 at 20:33
  • 2
    $\begingroup$ If you see worse performance in a linear regression after normalizing features, then you have a bug. The models on normalized and normalized predictors should give exactly the same predictions. A regularization term can affect this, but regularized models should always use normalized predictors. $\endgroup$ Mar 2 '16 at 21:11

One method that I like to use to see which feature contribute to a specific prediction is to reset all features to their mean one by one and then see how the prediction changes. I picked up on this method from this page. But I'll explain with an example of my own as well.

Say for example we have a model that predicts if a day is a good day to wear shorts based on some weather information, let's say temperature, wind and rain. and let's say we're using a method that gives us class probabilities.

Now we have a day where the model is predicting 50/50 for a given day, but we wan't to know what is causing this. So we will go through each of the features, reset them to their mean (or 0) and see what the model predicts now.

  • Say for temperature we have 20 °C, but the mean temperature is 10 °C. If we re-predict the model with the temperature for this day set to the mean of 10 °C, but keeping rain and wind at the same values, the prediction ends up being 80% for no shorts. Clearly temperature has a big effect! Now we can do the same for the other variables.

  • The wind speed is slightly above average, and by resetting wind to the mean and keeping the others equal, the prediction only changes a little to 55% for shorts. Seems temperature is a bigger deal.

  • Now rain is a bit of odd one, since rain is already on the mean. so resetting to the mean obviously would have no effect. But we still want to know if rain is influencing the prediction, so what we can do instead is set the rain to 0. And lo and behold, once we set rain to 0, the model predicts 75% for shorts. Again a pretty big effect.

By going through each feature and and setting them to their mean or 0 we were able to identify at a prediction level which features were important. Wind and temperature both had a large effect in either direction, whilst the wind had a much smaller effect.

Now why did we reset rain to 0? Doing this for temperature or wind would have been weird, since for these 0 is a value that barely ever happens and holds little significance, but for rain 0 is both relatively frequent and it means something specific, a dry day. Meaning it's sensible to reset to 0. So you really have to look at a feature by feature basis what makes sense.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.