What is the correct way to determine which features most contributed to the prediction of a given input vector?

Question

I am using logistic regression for binary classification. I have a big data set (happens to be highly unbalanced: 19 : 1). So I use scikit-learn's LogisticRegression() to train on 80% of my labelled data and then I validated with the other 20% (I looked at area under ROC as well as precision-recall because the data was so unbalanced; I also used the model with class_weight='auto').

My main question is the following: once I start generating predictions of unlabelled input vectors (using predict_proba()), how can I tell which of the features contributed the most to the prediction of that particular input? I imagine that this might be different than the "most important features" as determined in general for the model based on the labelled training data (e.g., coefficient magnitude).

I had a very basic idea:

1. Take the component-wise product of my input feature values with the absolute value of my feature coefficients. The most contributing feature is then the one that corresponds to the entry with the largest value.

2. Do (1) but use z-scores for everything (training and input features). I thought this would be important because I worried that some feature ranges might be very different from others and just taking products might not capture this; but I guess the coefficients should reflect ranges so maybe this doesn't matter.

Any thoughts would be greatly appreciated as I'm new to this. Things specific to logistic regression (i.e., sigmoid instead of just linear function) and any references to how to implement specific actions (e.g., transforms) in scikit-learn would be greatly appreciated as I actually am doing a project with real data.

Answer 1

There is a way using the regression coefficients only, you can understand which features most contribute to the prediction of a given input vector.

However you will have to standardize and scale each variable first (i.e. subtract the mean and divide by the standard deviation). Then refitting your model with the standardized & scaled data, the feature with the largest regression coefficient will be the feature that contributes the most to future predictions.

The regression coefficients are comparable after scaling because we have made the units of the features irrelevant, thus a one unit increase in feature $X_1$ corresponds to jumping 1 standard deviation of the unscaled feature.

Answer 2

One method that I like to use to see which feature contribute to a specific prediction is to reset all features to their mean one by one and then see how the prediction changes. I picked up on this method from this page. But I'll explain with an example of my own as well.

Say for example we have a model that predicts if a day is a good day to wear shorts based on some weather information, let's say temperature, wind and rain. and let's say we're using a method that gives us class probabilities.

Now we have a day where the model is predicting 50/50 for a given day, but we wan't to know what is causing this. So we will go through each of the features, reset them to their mean (or 0) and see what the model predicts now.

• Say for temperature we have 20 °C, but the mean temperature is 10 °C. If we re-predict the model with the temperature for this day set to the mean of 10 °C, but keeping rain and wind at the same values, the prediction ends up being 80% for no shorts. Clearly temperature has a big effect! Now we can do the same for the other variables.

• The wind speed is slightly above average, and by resetting wind to the mean and keeping the others equal, the prediction only changes a little to 55% for shorts. Seems temperature is a bigger deal.

• Now rain is a bit of odd one, since rain is already on the mean. so resetting to the mean obviously would have no effect. But we still want to know if rain is influencing the prediction, so what we can do instead is set the rain to 0. And lo and behold, once we set rain to 0, the model predicts 75% for shorts. Again a pretty big effect.

By going through each feature and and setting them to their mean or 0 we were able to identify at a prediction level which features were important. Wind and temperature both had a large effect in either direction, whilst the wind had a much smaller effect.

Now why did we reset rain to 0? Doing this for temperature or wind would have been weird, since for these 0 is a value that barely ever happens and holds little significance, but for rain 0 is both relatively frequent and it means something specific, a dry day. Meaning it's sensible to reset to 0. So you really have to look at a feature by feature basis what makes sense.