1
$\begingroup$

I trained a logistic regression model with 5 features per sample. Before training, I normalized the range of my features into [0,1] (MinMax scaler). After training, I received the following coefficients for a logistic regression model:

coef_1 = [[-2.26286643 4.05722387 0.74869811 0.20538172 -0.49969841]]

In logistic regression the coefficients indicate the effect of a one-unit change in your predictor variable on the log odds of 'success'. But as my features are normalized, I wanted to know the effect of a one-unit change in the original unit. For example, the range of the first feature is [0, 94.5] that of the second is in [0.5, 180], the thrid one is in [12,95] and the last two are categorical variables. So I divided the coefficients of the model by the feature range to get the 'real' one-unit change values and got:

coef_2 = [[-0.02393 0.0225 0.009 0.205 -0.499]]

I thought that these coefficients would be the same as I would get from a logit model where I did not scale my features before training, giving me:

coef_3 = [[-0.04743728 0.04394143 -0.00247654 0.23769469 -0.55051824]]

But these are clearly not the same. Although, there is some correlation. For the first two features, coef_2 is the half of coef_3, and for the last two features, one can say that coef_2 and coef_3 are approximately the same.

My question now are:

  • Which of the coefficients (1, 2 or 3) gives me the true change of a one-unit change in a predictor variable and how can I rank the importance of my features?
  • Can I even compare real-valued features with categorical variables?
$\endgroup$
2
  • 2
    $\begingroup$ I was working on a response, but I'm not going to leave it up until first asking why you standardized your variables in the first place and now want to change them back after fitting the model? This is an unusual thing to do, particularly in a logistic regression. What is the context of your question? What are the variables and what are you trying to discover? $\endgroup$
    – AJV
    Oct 12 '20 at 19:07
  • $\begingroup$ @AJV thank you for the feedback questions. I did standardize/normalize my features, not the final coefficients. I'm working on a model that will be predicting the chance that a shot opportunity results in a goal in soccer. The first features is the distance from the shot taker to the goal, the second is the angle in which the shot is taken, third is FIFA rating, fourth is whether or not the shot was taken with the player's preferred foot, and the final one is true if the shot was a header. I try to find which feature is the most important in maximizing the chance of a goal. $\endgroup$
    – Ruben_G
    Oct 13 '20 at 9:04
1
$\begingroup$

My question now is, which of the coeffecients (1, 2 or 3) gives me the true change of a one-unit change in a predictor variable

I've got several parts to my answer. The first is the most straightforward. You said you're using MinMaxScaler, so I'm guessing that you're using SciKit-Learn and doing this with Python. That package can make a scaler object that already has a method for undoing the scaling called inverse_transform(). There's more information over at stack exchange here or you can read the documentation here. I believe that your manual attempt at an inverse_transform simply didn't perform the right operations, and it is almost always safer to use pre-built well-tested functions from the kit anyway. It has the added benefit that if you'd like to see exactly how it works, it's open source--just read the code under the source link on the documentation site.

Part II may contain some things you already know, but here it is anyway. A logistic regression is non-linear, which means that the effect one-unit change in the predictor differs depending on the value of your predictor. The reason that we're allowed to make blanket statements in linear regression model interpretations, such as "for each 1 unit increase in $x$, $y$ tends to increase by such-and-such on average" is because a linear regression model has fixed slope coefficients. In other words, the first derivative with regard to any predictor is a constant, so the impact of a one-unit change is constant.

That is not the case in a logistic regression. As such, logistic regressions are typically used to predict the chance that a certain observation will fall into a certain category. The coefficients in a logistic regression are not regular slope coefficients that can be interpreted as simple unit-changes as in linear regression, they are logged-odds ratios. This makes logistic regressions much less intuitive to interpret. That much you already alluded to in the question.

However, the reason for performing the transform to begin with is unclear. If you had serious multicollinearity, sure. If the spread of the values was huge, maybe it makes sense. If there were several different units, like kilos, pounds, and inches all being used from different systems that measure different features, again, it might make sense. But I'm not convinced you really need to normalize your variables to begin with in this situation. I'd recommend trying the following instead:

  1. Do not standardize your variables before you fit the model
  2. Exponentiate the coefficients you get after fitting the model. This will convert them to odds instead of logged-odds. If you want, you could further convert them to probabilities to make interpretation even easier. The formula is $$probability=odds/(1+odds)$$
  3. Interpret it as "for each [insert unit] increase in $x$, there is a such-and-such increase in the odds (or probability) of $y$ occurring.

how can I rank the importance of my features?

I'm not sure if you need a formal test of this. If you've got the odds or probabilities, you can use your best judgement to see which one is most impactful based on the magnitude of the coefficients and how many levels they can realistically take.

Can I even compare real-valued features with categorical variables?

Sure, but I wouldn't normalize them first. You can compare different types of variables to each other, just bear in mind the meaning of the different types. If increasing the distance from the goal by 1 meter decreases the probability of making the shot by 1% and having good weather instead of bad increases the probability of making the shot by 2%, that doesn't mean that weather is more impactful--you either have good or bad, so 2% is the max increase, whereas distance could keep increasing substantially and add up. That's just an example, but you can just use common sense like this and explain your reasoning when you're comparing them.

$\endgroup$
1
$\begingroup$

I agree with AJV's response, big idea is the linear equation is related to the log odds:

$$\frac{log(y)}{log(1 - y)} = \beta_0 + \beta_1x$$

So for Interpreting the Coefficients; SKlearn will give you the coefs and intercept as per usual after fitting a model, but the equation above is what they relate to. Thus, out of the box our logistic regressor returns log odds.

From here, it isn't much work to turn odds into a probability which is what we would get solving for y.

Finally, we use this probability to make the classification -- less than 0.5 we will say it is class 0, greater than 0.5 it's class 1.

Sklearn gives you all of these for a point or array of points with a fit model. For your example

In sklearn, you would get all these after fitting a model; for example:

clf = LogisticRegression()
clf.fit(X, y)
clf.predict_log_proba(X) #gives log odds back
clf.predict_proba(X) #converts log odds to probabilities
clf.predict(X) #uses the probabilities to make predictions

Now to your second point sklearn uses regularization by default. From the documentation:

Note Regularization is applied by default, which is common in machine learning but not in statistics. Another advantage of regularization is that it improves numerical stability. No regularization amounts to setting C to a very high value.

Thus, if you want to not worry about scaling set the C value high.

With scaled features, we still won't interpret our coefficients as we would with linear regression so we aren't really in any worse situation. A bonus of scaling our features here would be that we can compare the relative importance of the features whereas we would need to do work if we hadn't scaled our data. For more on this see the example in sklearn's docs here.

$\endgroup$
1
  • $\begingroup$ Nice to see you here, @jkhoehler. You can set the scikit-learn LogisticRegression penalty to none to avoid regularization. $\endgroup$
    – jeffhale
    Apr 24 at 13:37
0
$\begingroup$

I feel that the discrepancy has something to do with the non-linear nature of the logistic regression model. If you use a linear model, what you did might give you what you expected.

coef_1 can be interpreted as the change in log odds ratio corresponding to 1 unit change in standardized X.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.